I'm going to refuse to talk to my brother
Why don't you want to talk to your brother?
He's a downer and he's constantly bitching about me and talking shit about me.
Brothers can be annoying but you shouldn't stop talking to them. It won't solve your issues and will lead to a worse relationship.
I'm going to do it anyway
Don't you think you'll miss him if you stop talking to him altogether? Maybe you should consider just talking to him less, but not altogether.