I think I am only going to talk to my grandparents (who I barely know but still love) every few months
Why are you only going to talk to them every few months? Wouldn't you want to spend more time with them?
Because neither of them spend time with me and they go on trips and do what they want, and it irritates me.
Your grandparents have their own lives and are busy people. You should do your best to keep in touch with them, and when you do talk to them, you can suggest ways for you to spend more time together. If you invite them to do something, they might return the favor!
I can't be bothered to keep in contact with them. They aren't that important to me.
It sounds like it's one-sided. I don't blame you for limiting your conversation or visits because they should also care a little bit more about you too. I definitely see where you are coming from, but nonetheless, they are still your family, so do the best you can.