My friend keeps asking me to pay him what I owe, but I am not going to.
That's selfish of you. Why won't you pay back your friend?
Because one he annoyed me by constantly sending me text messages.
That's not a good reason not to pay back your friend. You should always pay back your debts, especially to friends. Not doing so will just cause that relationship to deteriorate.
I already forgave him for bothering me with all those texts. He should just be grateful I did that and forget about the money.
It's good that you forgave your friend for annoying you, but that doesn't change the fact that you owe them money. If you're going to borrow money and never pay it back, people will not trust you in the future. It's likely you friend won't want to remain friends with you if you act so selfishly.