I'm going to turn a guy down because he's short
What are you thinking for turning them down for their height?
I'm very tall and I feel uncomfortable when I'm with someone shorter than myself.
Everyone has their own preferences in what they find attractive or not in another partner and it's okay for you to have preferences of your own. If that preference is height-related, that is perfectly okay as you shouldn't feel compelled to find someone attractive that you do not.
People who are worth dating should have many desirable qualities and height should be one of them.
Definitely, you shouldn't have to date people you aren't attracted to for the sake of not offending anyone. It is your life, and you need to stand up for what you want! You in the future will thank yourself for choosing someone you have the preferences for.