I really don't want to invite a friend of mine to dinner because of his limited diet.
Why does your friend have a limited diet?
Because he's on a diet and wants to lose weight
I understand why that might be annoying, but it's rude to not invite your friend just because he doesn't eat very many foods. You should try to be accepting of his choices, and still invite him to dinner or he might not want to be friends with you any more.
I'm not going to do it, he deserves to die from hunger.
You shouldn't discriminate against your friends eating habits. You should be happy for him trying to lose weight.