I want to cut contact with a friend who is down on his luck
Do you think it's a good idea to stop being someone's friend when they are down?
It's his own fault. He is a bad friend.
It sound like you might be the bad friend.  If he has been down on his luck he might be having trouble being a good friend right now but that doesn't he deserves to be ditched.
You can't judge my character
How would you feel if he abandoned you when you were having a bad time? You should consider his feelings.