I want to roleplay in dungeons and dragons in my downtime,  but the dm wants me to spend my downtime outside of their game
Do you think it is fair for the dm to ask that of you?
It's unfair and I do not need this bullshit.
It doesn't sound fair of the DM.  He should be open to letting you participate along with everyone else, as long as you are following the rules.
I'm going to go spend my money somewhere else.
That sounds like a good idea.  While I think that the DM should have allowed you to join the game, if they will not welcome you, then you should take your time and money elsewhere.