I'm thinking about cutting off most contact with my grandparents
Why would you cut off contact with them?
I can't stand them being so frustrating and so aggravating
If people are truly toxic to your well-being, then it's okay to cut off contact with them, but it sounds like your grandparents just don't realize that they are upsetting you.  It is very rude to cut off contact with anyone without telling them why, and especially with family members you should be willing to forgive them.
My grandparents can die for all I care.
What a terrible thing to say about your grandparents. You should stay in contact with them as much as possible because they might not be around much longer.