I think I'm going to leave my friend behind.
Why are you going to leave your friend behind?
I feel like he keeps holding me back.
I understand how you feel, but it's not right to leave your friend behind. It's wrong to just abandon him. How would you feel if someone who you cared about just stopped talking to you?
It's none of your business.
While it is good to stick with friends, it's also understandable that you may want to end a friendship that you feel is preventing you from growing as a person.  Just make sure that you're ending the friendship for a good reason, otherwise you may regret losing your friend.