I'm going to abandon my friends because of one weird kid
Why would you abandon your friends because of one person?
I really don't want to hang out with that kind of person
It is fine to not want to hang out with someone. You don't need to hang out with people you don't want to.  But you shouldn't be abandoning everyone else, as well
That is what I have been saying.
Tell your friends you do not feel comfortable hanging out with him. Your friends should understand.  Maybe you can set up separate activities so that everyone can be happy?