I'm thinking of ghosting a friend with severe anxiety issues
That feels a little cruel, why would you want to ghost a friend with severe anxiety issues?
He really annoys me when I'm trying to play a game I enjoy.
I'm sure that you would like some alone time when you're playing a game you love, but it's wrong to abandon your friend because of that. Your friend would be confused and sad if you ignore him and it won't help him with his issues. Try to explain to him that you need some space and that there will be enough time for the two of you to spend together.
Nah, I'm not really trying to reason with a crazy person.
It's not very kind to call someone with anxiety issues as crazy, wouldn't that make him feel worse? How about letting him watch you play, and you can discuss his issues while you play! I'm sure he'd appreciate your company in any way.