I gave my small son 20lb weights to play with while I worked out
Why would you give weights that heavy to a small child?
It's a little bit of fun for my son.
You have a small son that you want to keep busy and make sure he has fun, but weights that heavy can harm him. Be prepared to have to call the emergency room after this because he will get hurt. I strongly advise to not give him the weights at all to prevent such an ordeal in the first place.
So I should punish my kid for having fun?
Absolutely not, it's great that your child wants to have fun when you work out. I just suggest giving him smaller toy weights that are appropriate for a child his size so that he doesn't hurt himself.