I'm going to leave my family vacation early because of my brother.
Why? Is he doing something to bother you?
He keeps annoying me and making me feel bad.
It's reasonable to leave your family vacation because you don't like being around your brother. You don't have to stay somewhere that makes you feel uncomfortable. I recommend talking to your brother before leaving, though, because you might be able to work it out.
I don't think that would help. But thank you for the suggestion though.
I would still urge you to communicate though, as it may simply be misunderstanding that's why he's acting that way. If you don't talk with your brother, then he may not understand what he's doing to you is wrong, so it's better that you tell him.