I'm mad every time my SO tells me that I'm crazy
What's the reason why he would say you're crazy though?
He never listens to me and he's always talks down on me.
If you feel that is he might be treating you unfairly, then perhaps the two of you have to discuss things in private. It sounds like the both of you have things to work on with your relationship.
No way. He needs to take responsibility for his actions and admit that he's wrong.
I agree that calling you crazy and talking down on you is wrong, but I feel like saying that you have no responsibility for it seem a little bit dismissive. You may have provoked something to make him say those things and it is important for you both to accept mistakes and find a solution on how you can have a better dialogue between you two moving forward.