I don't want to try doing things with my friend anymore
Why don't you want to do anything with them?
I always feel left out when we do stuff with other people.
You should talk to him and tell him about the situation about feeling left out. Maybe your friend doesn't see it your way or you are just seeing it wrong.
I think I'm going to tell him that he is a jerk anyway.
You don't have to call anyone a jerk, just tell them you feel left out sometimes and I'm sure they'll understand.