Can I uninvite someone to dinner because they are vegan?
Why would you do such a thing?
I don't feel like making a special meal for them
It is rude to uninvite someone after the plans have been made because of dietary restrictions. It is more friendly to make an effort and then not invite them next time.
I'm not veg so they can go fuck themselves
Do you want to be the type of friend that is known for disinviting people because of their meal preferences? Some people are Vegan because it's the only way their body can handle food, uninviting them is basically telling them you don't care about their health.