I'm going to avoid interactions with my boyfriend's brother's new girlfriend
Had you already planned on seeing her somewhere?
I'm hosting a party and I don't want her there.
That choice might cause some conflict between your boyfriend and his brother.  But if your mind is made up, you should be honest with them and tell them exactly why she isn't invited.
I'll just say she's not my best friend.
If this party is meant to be a small and intimate affair, then you can say that only a small number of people are invited, and everyone will understand.  However, if you want to exclude your boyfriend's girlfriend from more casual get-togethers, that could cause hurt feelings.  It might be best just to avoid her when you can, and accept her presence at other times.