CAMEL-21269: camel-langchain4j-tokenizer fixes for the YAML DSL

[orpiske]

This is a more more comprehensive fix for the tokenizer support for the YAML DSL. There were quite a few problems in the model files that prevented it from being used with YAML (thus affecting any sub-project or project relying on this).

Although it's relatively big due to the regen of a few files, all the changes are backwards compatible and anyone using it with the Java DSL should not have to change any code.

[github-actions]

🌟 Thank you for your contribution to the Apache Camel project! 🌟

🤖 CI automation will test this PR automatically.

🐫 Apache Camel Committers, please review the following items:

• First-time contributors require MANUAL approval for the GitHub Actions to run

• You can use the command /component-test (camel-)component-name1 (camel-)component-name2.. to request a test from the test bot.

• You can label PRs using build-all, build-dependents, skip-tests and test-dependents to fine-tune the checks executed by this PR.

• Build and test logs are available in the Summary page. Only Apache Camel committers have access to the summary.

• ⚠️ Be careful when sharing logs. Review their contents before sharing them publicly.

[orpiske]

This PR is ready, but I need to wait until Monday/Tuesday for a review from @lordrip on the resulting YAML on Kaoto's end. I want to play safe because I need to backport this to 4.8 as well.

[davsclaus]

You should also ping @mgubaidullin as it can affect Camel Karavan

[orpiske]

You should also ping @mgubaidullin as it can affect Camel Karavan

Absolutely. I couldn't reproduce w/ the existing version of Karavan, but even if it got everything correctly, the generated YAML would still be wrong.

It's on my TODO list for next week to figure out if/how Karavan is affected so I can pinpoint exactly where/how we need to fix any code relying on the incorrect model.

[orpiske]

I've just confirmed that the YAML itself is functional. Now I'll work with the users / tooling creators to make sure they can leverage the new EIP.

Let's merge this to make it easily available for others.