I don't want my housemates partner around
Why don't you want them around?
I don't like to have him in my house.
Unless he is breaking the law or the rules of the house, your housemate should be able to have him around. Your housemate is living there and can do as she pleases.
Fine. I do what I want and they can be mad at me.
You should not want the people you live with to be angry with you. You have to deal with him being in the house, try and at least get to know him better.