I'm thinking of backing out of a trip me and my brother had planned since last year
Why would you want to skip it?
I'm so sick of all his complaining
It is hard to get along with siblings sometimes but you shouldn't cancel just because he was annoying. He already made arrangements and it would be rude to cancel things now.
You are so annoying, I'll do whatever I want.
He might be really upset if you cancel it. You could lose your deposit if you back out now. Have you told them how you felt about all the complaining yet?