I was thinking and I don't want to associate with my brother in law
Why don't you want to associate with him?
I don't like him and he's a general pain in the ass.
I can see why you wouldn't want to be around him. It's fine if you would rather not associate with one of your in-laws.
I don't want him to come to my house anymore.
it's okay to not get along with everyone in your family, and it's your house and your rules. Are you able to talk to your husband about this situation? It would be great to have a compromise and he can just visit your in-laws instead.