There is a family gathering that I want to decline.
What is your reason for not wanting to attend the gathering?
My grandpa always bugs me at gatherings and I get tired of it
I understand that your grandpa bugs you, but you shouldn't skip the family gathering for that reason. Your grandpa might annoy you now, but you may regret not seeing him later when he is no longer around.
Family gatherings get on my nerves.
I completely agree that family get togethers can be a pain, but it's disrespectful to skip them and let down the people who expect you to be there. It's just one gathering and you might end up having a good time.