I'm not going to pay back the loan I got from my friend.
Why wouldn't you pay him back if you owe him?
He makes disturbing comments that I find that inappropriate.
He may say some disturbing things, but you did borrow the money. You must repay the money he loaned. After that, you can just ignore him.
If I don't pay him, all he'll do is tell me I'm a bitch and mess with me. It's not like he can hurt me.
He might not be able to physically hurt you, but he could sue you for the money if there's any proof it was a loan, even just a text message. Then you would be embarrassed in court and to anyone who knows you both, and you'd also be out the debt money and the legal fees. It's better to repay your debts and be done with it.