I'm going to frequently turn down my friend's invite to tag along
Why would you turn down your friend's invite?
I think going out with him is a drag. I hate going places with him more than others.
I see your point, but it's pretty rude to ignore social invites. You're friend might be upset if you stop hanging out with him.
Wah, I'll be really sad I if don't have any friends
Then I believe it's a good idea to just let your friend come along still. You never know if you're going to have good experiences with them, and broaden your friendship!