I'm thinking I made a mistake and should ditch this guy at once
Why do you want to ditch the guy?
He... is... so... disagreeable to be with.
It's pretty mean to ditch people if plans were made especially going out with a group. You should consider how he feels about it and if you don't like him, tell him. It's not right to ditch people and you might look bad to other people for doing it.
I didn't want to go out with him in the first place, I only said yes because of peer pressure.
Well that was your first mistake. You shouldn't do things just because people are pressuring you. That's how you end up in bad situations like this one.