I really don't want my extended family at my graduation. Would it be OK to tell my mom not to invite them?
Hm, why don't you want them there?
I had some bad experiences with them making me feel uncomfortable on my graduation day.
It's understandable to not want your entire family to come to graduation. Your graduation day is a day to celebrate yourself and having family members that you don't like would only distract from that. Find a polite way to decline having your extended family come.
I just don't want to see my extended family on my important day, I don't want them to ruin my day.
It's up to you whether you want to have them there or not. Either way, there's no shame in not liking people in your extended family.