I don't want to visit my brother
Is it because of something that happened to you?
Yes and I don't want to remember it, its too painful.
I respect that you don't want to talk about it.  And it's totally cool to not want to see family members from time to time.  Just reach out and let them know you won't be coming so they don't wonder what's happened to you!
I will let them know when the time is right and I will keep ignoring them.
If you are having problems and mental anguish with what happened have you tried talking to a therapist? I think it might help you try to work out some of your issues to talk to a neutral person.