I hang out with some homeless people. I don't want my friends around them.
Wait, why? Are you ashamed?
No, I just enjoy going out with homeless and spending time together.
At the end of the day, you can't control your friends. You should not try to tell them who they can hang out with.
I guess you're right, I just don't want them interacting with us.
It seems like you're being secretive, which is fine, but you simply can't tell your friends what they can or can't do.