I thought I was the more startled one until he sprayed me
Is that why you think he sprayed you?
Yes, I think he was scared and that's why he did that.
It sounds like you learned your lesson. Skunks spray people if they are startled and you should try to avoid them because of that.
I just wanted to try and pet him.
That's probably exactly why he was startled. Approaching wild animals will scare them and could cause this to happen again in the future. I'd suggest taking a bath and trying to get rid of the smell.