I would prefer for my kids to not call him grandpa to be honest.
Did something happen in the past?
My dad just always talks crap about me. He says that i was the worst kid to take care of.
Just make sure to be present around your kids when your dad is around. Talk to your him about the past and hopefully he will apologize and you both move on.
I think he's going to say something to my son about how I was a mistake.
If you do not want your father around your kids because he is going to talk bad about you, than that is your choice. You are free to choose whoever you want to or who you do not want to spend time around your kids.