Is it wrong to want my ex to move out of my house?
Why would you want your ex to move out of your house?
Because he is annoying and leeching.
If you want him out of the house you own, then that is your right as a homeowner. You do not have to put up with him if he is not being respectful of you and your house.
This is my parents house though so I may not be able to?
While it isn't your house, it is your parents home and they could kick him out for you if you need him to be kicked out. It is your house more than his and he should have to find another place to live if you aren't still together.