Is it okay if not going to try harder to make time to game with my friend?
Why don't you want to game with your friend?
Seriously, he is a pest. I need a break from this guy.
Even if he annoys you friends are expected to share time together. If you want to have them as a friend then you should make the time to game together.
Okay, I will tell him to leave me alone.
You could try to set a certain amount of time on certain days of the week to better manage when you and your friend would hang out gaming. Your friend wants to have fun with you, so try to appease them a little bit.