I eventually caved and gave him my notes
How did he feel about that?
He was still mad that I did not give my notes sooner.
You should do your best to stand up for yourself. Caving into peoples demands will only mean that more people know that you're not strong and willing to cave given enough pressure.
Now because I gave them late, he wants to ruin my reputation.
I am sorry that it happening to you. Can you talk to him and ask him to not say anything.