Is it okay for me to be man and afraid to fight?
I'm not sure. Do you walk around trying to fight people?
I will kick back if someone insults me, but I never start a fight
That is okay not to fight. Not every man has to be a testosterone driven juicehead. I respect you for that!
But they don't respect me. I want everyone to see how good I am at fighting but I'm afraid.
I believe you don't need to try to impress people who don't respect you. Fighting can really hurt and mess you up, so it's perfectly fine to not want to do any of that. How about just ignoring them and doing what you love instead?