My friend wants to join me on my own vacation and I don't want to invite him.
Why don't you want him to join?
I want to spend some quality time with myself and get the chance to explore my inner self.
Just tell your friend you prefer to go alone. There is nothing wrong with going on vacation alone and getting some time to decompress. If you tell your friend he will respect your decision.
I feel like I have no choice but to say yes to him because we are friends.
That is not something you should do here. You want to go by yourself so you should. If he is your friend, he will understand your request and not go with you.