Would I be a horrible person for backing out of my buddy's wedding
Why do you want to back out of your friend's wedding?
I've been feeling less and less comfortable being around him recently
If you agreed to be in the wedding then you need to fulfill your responsibilities. You are being a terrible friend if you do back out of the wedding.
I'm over that too. I'm unwilling to waste 2 hours at the wedding and all that jazz.
You really shouldn't do this to your friend. He was counting on you to be in the wedding and you're putting him in a tight spot to replace you. Doing this could mean the end of your friendship.