I always use the handicap stall but only at the gym
Why take up the handicapped stall if you are not handicapped?
I go to meet my friend after the gym class, he is handicapped.
If you are saving it for him, then that is fine I suppose. It sounds like you might be using it too, which would be wrong as you are not handicapped yourself.
I have no problem with leaving it empty as long as there are others who need it more. Usually they are working out so I can use it when needed.
It's true that the stall will not be in use all of the time, but the fact of the matter is that you don't know when someone will come and in need that stall.  Since you have a friend who is handicapped, you should appreciate how difficult it is for some people, and you shouldn't inconvenience them, just in case they came in while you were using the stall.