I would rather not be friends with this boy who was actually abused.
What is your reason for not wanting to be friends with him?
I don't like his mannerisms because he always appears like a victim and no one likes a whiner
You are only perpetuating his victimhood by bullying him like this and refusing to be his friend. All he needs is someone to be there for him and support him, and you're only making it worse by talking down on him and hurting him.
Those comments were just jokes.
This is not really something to joke about. Someone who has been abused probably really needs some friends in their corner, so to joke about not wanting to befriend him is kind of shallow.