Is it wrong that I'm not going to pay for my adult son's youth conference?
That depends on the situation. Why don't you want to support your son?
For the past 20 years, he's been such a good-for-nothing. It's only my fault that he doesn't have to work hard
I understand how you're trying to teach something about responsibility and it is definitely your right to not pay for something that your son is involved with, but I sure do hope that you explicitly tell him that it's his responsibility to pay for that conference that he's attending and not yours. Your son is an adult now, and he shouldn't expect you to pay for something that he is responsible in paying.
Maybe when he has a job, his own money. Then I will tell him to pay for it.
It seems you've had a change of heart when it comes to paying for it. Just be sure to talk to your son about being responsible and starting to support himself. Without your guidance your child may be unaware of his potential in life and responsibility in life.