I want to drop a friend
Is there a reason you want to drop him?
I feel like quitting him because he's always been a dirty fucker.
You are not obligated to be friends with anyone, so it's completely fine to dump a friend who is not a good person.  If you tell him you don't want to be friends in a cordial way, he should understand without causing to much drama and stress.
I refuse to pick up the phone when he calls me.
It's understandable that you don't want to talk to him at all. I just think you should maybe answer once and tell him you are no longer friends.