Is it wrong that I'm going to press the close door button instead when someone asks me to hold the elevator?
why would you press the closed door button?
I'm in a hurry and I don't feel like saying an uncomfortable no.
You shouldn't shut people out of an elevator. You could meet that person again and they could shut you out.
It's not like their genitals are going to catch on fire.
It will take an extra minute, if that, to hold the door open for someone. Letting it close would be rude. It's best to remember to treat others the way that you want to be treated.