Do you think it's rude that I always tell my coworkers family that I'm busy when they invite me over?
That doesn't sound right. Are you actually busy?
Depends on what you mean by busy. I could be playing games, going to the club or even having sex with some guy.
It is rude to tell people you are busy if you just want to avoid them. You should not play games with people like that.
Who cares I'm having a great time.
If you really don't want to see them then you should just stop wasting their and your time and tell them. It's rude to just make it seem like you're busy to avoid people.