He demanded to know why I was looking at him.
That's fair to ask. What did you think of telling him?
I'm going to say that I want to check him out
It's nice to look at attractive people, but full on staring is a no-go. You are putting the unwanted spotlight onto them and you can make them very uncomfortable.
No one complains about guys looking at attractive women, why do you care about me looking at guys?
No matter what gender someone is, I believe that it is not kind to stare. Men have feelings too, and don't you think it's unfair to think that rules don't apply to them? How about coming up and talking to them instead, that is more welcoming.