I'm going to tell my friend I don't want to get lunch with his friend
You don't like his friend?
No, I don't want to get lunch with his friend because I never have the time
It's fine to not want to go to lunch with certain people. However, it could end up hurting his feelings.
He'll get over it.
You can just explain how busy you are. It can be hard to make time for socializing.