I'm going to tell my brother to stop bringing so many people over for family dinner
Do you really have a problem with this?
If I didn't have a problem with it, I wouldn't bring it up.
Unless you are hosting the family dinner, you shouldn't tell your brother who he can or cannot bring.
I hate being around so many people.
You should ask your parents if you want to have people over yourself.