I'm drifting away from a friend who i going through a hard time.
Why do you think you're drifting away from your friend?
I don't think I want to deal with his problems anymore.
You shouldn't abandon a friend who needs you. Maybe you can check in with your friend and see what type of support they need? Your friend might have a harder time if they feel like you're abandoning them.
Maybe he really doesn't need my support. I don't want to bother him.
I understand not wanting to bother a friend, but he won't be bothered if you are checking in to see how he's doing. I'm sure he would appreciate the offer of support since that's what friends are for!