I'm going to tell my friend that I'm not spoiled
Go for it. Does he think you're spoiled?
He says that I'm the only one who has never done any work.
Telling him you're not spoiled doesn't make it so. Let him think what he wants, as only your opinion about yourself should matter.
I'm a good worker.
If you are, then you should be proud of what you do and not have anyone else's opinion try to shift you. You friend can think whatever he wants, but if you think it's worth the effort, show him how much of a hard worker you are!