I'm a little bit annoyed that my brother still won't forgive me for being a bit of a bully as a child
Why do you feel that you deserve forgiveness?
I feel like he should be the bigger person and accept my apology
The things you've done to him may not have healed yet and it starts with a sincere apology. Try analyzing your apology and his reactions and see if that was a good apology as it may be a bad one.
I can't apologize to him because I don't feel bad about it.
That's a more important problem - the fact that you aren't sorry for what you did and expect your brother to get over it is unreasonable. His healing will start when you acknowledge that what you did was hurtful. Then maybe he will feel like he can forgive you.