A kid asks his dad why people who farm are known to have green fingers because they are normally brown
Is that supposed to be about black people?
It's just a joke
Joking about black people being forced to work the farms is not tasteful or funny. It is hurtful to be reminded that your ancestors were forced to work as slaves.
Who cares about their pain?
Many people, and some people will think that you might be racist if you are okay with telling jokes like that. They might not want to be associated with you.