I don't want to play d&d with an annoying roommate
Why don't you want to play d&d with him?
He really is too much for me. I'm sorry but he writes the dumbest novels.
Well, regardless of your reasons I don;t think that's a good idea.  He might be upset with you if you leave him out of an event like that which he'd expect to be included in.  This might lead to some friction in the house moving forward!
No worries even I think it's a stupid idea.
So you're going to play d&d with your roommate now? Well I'm glad!