I think short guys should be referred to as 'boys' not 'men' regardless of age
That's kind of an odd way to think about short people. Do you think this is okay?
It does sound 'funny' to call an old man a boy, but it's fine.
You should not judge people based on their height. It's rude to call short men "boys."
I'm not judging but it's the truth.
There's nothing inherently unmanly about being short, and it's rude to imply that there is. If you do this very often, you'll find that people get offended and don't want to spend time around you.