I was punched and never even defended myself.
Do you feel like you are unable to standup for yourself?
He has beat me up several times before. I know this is what he really wants, I don't want to make him angry.
It's wise not to fight fire with fire, I get that. You shouldn't just let yourself get abused, though. Call the cops or let someone know.
I got tougher in jail and now he is not harming me as bad.
It is a good idea to keep from using violence. However, when being bullied it can be helpful to show the bully that you will not tolerate allowing someone to physically harm you. May be a good idea to enroll in some sort of combat class or one that is for self defense.