Do you think it would be appropriate for me to kick my son's mom out of my house?
Why would you kick her out?
She had an affair with the neighbor on my bed.
You don't have to tolerate people in your home who you don't want in it, and what she did is certainly not something you should have to put up with. You should ask her to leave.
What if she doesn't want to leave?
If she doesn't want to leave you need to remind her that it's your house, and you're no longer comfortable with her living there. If she isn't paying rent then she has no right to live there if you don't want her to.