I used to bully my younger brother as a kid and I hate myself for it.
That's horrible! Why did you bully your brother?
He would annoy me so I would want to get revenge on him.
It's not okay to bully your siblings. If you feel remorseful about it, maybe you should make it up to your sibling by apologizing. It might ease any strain in your relationship.
I don't even know if he would forgive me if I did apologize.
Whether he accepts your apology or not, you should still apologize to him. That way at least you know you did what you could to try to make amends with him and you can be at peace with it. It can only help your relationship to tell him you're sorry for how you treated him.