Can I tell my mom that if she invites my brother to the wedding because she wants him there, that she is responsible for him?
Why would she be responsible for your brother?
I don't want to deal with my brother, I would want my mom to just be responsible for him to treat him nicely.
I know that being responsible for the wedding can be a very overwhelming task, but your mother may already be responsible for a lot of different people at that event, and maybe very burdensome for her to also be responsible for your brother. How about you speak with your brother, and try to talk with him that if you are gonna be inviting him, that he would be on his best behaved self.
My brother is a dick, why should I care about him?
He is family and it is expected you invite all family members to your wedding regardless if you like them or not. If he is causing trouble then he can be asked to leave.