It is hard to let go of anyone.
Really. Why is letting go so hard for you?
I don't want to lose anyone who matters to me.
It is nice that you care about others.  Sometimes, you do have to let go though.
But I don't want to lose them.
But if you care about them then you should let them make their own decisions if they don't want you to be part of their lives. It is good to care about others and that includes their feelings.