I'm a smoker and refused to move out of my office for an asthmatic colleague
Out of curiosity, are you smoking inside your actual office? Generally that is prohibited nowadays.
Yes, but I don't want to move since the guy is also annoying all the time
I completely can relate to having an annoying coworker, but you shouldn't smoke inside your office. If you have an issue with them, perhaps you could have a civil conversation with them or discuss it with HR. In the meantime, I'd suggest going outside to smoke.
What else am I supposed to do?
Just be mature and mindful of how your actions impact the health and well-being of others, even if they are annoying.