I know it tears my parents apart inside that I refuse to have a relationship with my brother, but I honestly couldn't care.
Why don't you care about how your parents feel?
I'm not gonna put myself through grief for a brother that don't deserve my attention.
Relationships with family members can be a tricky thing, and you don't always need to do what your parents want. However, you should try to think about your parents' feelings. You may regret keeping a family member out of your life later on in your life.
Screw that. They can deal with him.
You may have your reasons for keeping your brother at arms' length, but unless it is for very good reasons, it is time with family you can never get back. Do this for your parents, not your brother.