Is it okay if I say I'm not racist but I only say racial slurs at minorities in acceptable situations?
Wow, seriously? If you say racial slurs at minorities in acceptable situations as you said, then you are a racist.
I wouldn't say that I'm racist, but I've grown up around racists
You should never say any racial slurs to anyone no matter what the circumstances may be because that makes you a racist. So if you say you are not a racist then you shouldn't make a racist comments or else you look and sound like an ignorant hypocrite.
I know but sometimes I think that minorities should just get over it if I do it in a joking way.
Racial slurs are truly not acceptable in any situation, not just when you think it's appropriate. That's not how it works, either you are or aren't. Saying nothing is better than saying something negative.