I don't want to spend time with my brother
Is there a reason that you do not like to spend the time with him?
He steals my stuff and he is such a pain in the ass
If he steals your things, you don't have to spend time with him. That's completely okay.
He has some nice stuff.
I think that you should protect yourself from him. I think that it is okay to chose to not spend time with him if he just steals from you anyways.