I have no respect for my manager
Why don't you respect your manager?
He is rude and doesn't let me have the days off I want.
If you openly disrespect your manager by refusing to do your work, you can expect there to be consequences. You might find yourself fired for refusing to work. It would be best to keep your feelings about the manager to yourself and just do your job well.
I'm still going to talk shit about him to my coworkers
It sucks when your manager isn't competent enough to treat others kindly, but that is usually apart of every job. You just have to put up with it and hope your manager changes or start looking for a better job with better bosses.