I don't want to invite extended family to my college graduation
Why don't you want to invite your extended family?
Because I'm tired of having them in my way all the time
It's fine if you don't want certain people at your graduation. This is your special occasion so you should get to choose who takes part in it.
All I know is that every time I invite them, they ruin everything for me.
I think you just shouldn't tell them about the graduation at all to avoid conflict.