I have the worst relationship with my brother. If it weren't for us being brothers, I wouldn't even talk to the guy.
How does he treat you?
My brother used to hit me until I was sick in the hospital when I needed an operation.
It's perfectly fine to cut toxic people out of your life, even if it's your brother. Otherwise they could stay in our lives just to torment us.
It's not any better now that we are grown up.
If your brother isn't changing, and you've tried talking to him about his behavior, then it seems like you might have to cut him out of your life. At some point, you need to focus on what's best for you and end the relationship, even if it's someone as close as your brother.