I'm not going to spend time with my brother when he's sick
That seems reasonable. Why does he want you around when he is sick?
I'm not going to spend time with my brother when he's sick anymore because I don't want to get sick.
I think it's smart of you to avoid your brother when he's sick, you don't want to catch anything and spread it to others.  If you tell your brother you don't want to get sick, he should understand and support your decision to avoid contact.
He's a huge baby and I hate him anyway.
It sounds like you're angry with him about something unrelated. I understand that you don't want to get sick but if he needs someone to care for him you might want to help if you can.