I hate it when people end up tagging along on my traveling plans
Isn't it fun to have company? Why do you hate that?
They weren't invited to begin with.
You have a point. It's not nice to just invite yourself on vacation. It's expected to plan these things and talk them through first.
If they want to travel, they need to deal with the planning.
I agree it's rude to invite yourself on someone's trip. You should turn them down the next time someone tries to do that to you. You shouldn't feel bad about wanting to enjoy the trip that you planned.