I'm going to refuse to go out with my co-workers
Why don't you want to go out with them?
I am not a fan of where they are going to.
If you don't like where they are going, then you don't have to go. Do not force yourself to do something you do not want to do.
Great, I will go home then.
There is nothing wrong with that. Maybe you can suggest a place you like to go with them next time.