I'm going to refuse to spend time with my grandma. I just don't want to do it.
Why would you refuse to spend any time with your grandmother?
I don't want to be a grandma's boy.
Refusing to see your grandparents just because you dont want to be labeled as  a grandmas boy does not make a lot of sense. If you enjoy spending time with your grandmother than go ahead and spend time with her. Anyone who is calling you a grandmas boy is simply putting their own insecurities onto you.
Okay you are right. I will spend time with my grandma.
A lot of people don't have their grandma's anymore and its great that you still do and can spend time with them.