I'm not going to invite a friend to my birthday party
Why wouldn't you invite your friend?
I don't like him? No further reason.
You have the right to not invite someone to your party.   If you are seen as friends with someone but do not like that, that is a larger overall problem.  You should either try to patch the friendship up or gracefully end it on order to avoid bad blood between the both of you.
But they can't hurt me and they can't go against me. I'll do as I please.
I'm just saying that if they are really your friend then you should invite them to your party.