Can I just not respond?
What are you not responding to?
This conversation.  I'm sick and tired of conversations like these.
It's bad to just stop a conversation arbitrarily like that.  You should respond when prompted!
Why don't you go away?
You're acting in a way that most people would call extremely rude. You shouldn't just ignore someone, and you definitely shouldn't tell them to go away. Try being a kind host and eventually the conversation will end naturally.