KAFKA-16237: Add Scala 3 support for Kafka Streams

[mberndt123]

A port of Kafka-Streams-Scala.

The code itself was almost entirely compatible, most of the work revolves around the build system.

Because core doesn't yet support Scala 3, it's necessary to specify the Scala version for the two separately, so there is now a streamsScalaVersion in addition to scalaVersion.

The two versions need to be compatible though because the Kafka-Streams-Scala tests have a dependency on the Core project. Fortunately Scala 2.13 and Scala 3 can coexist on the classpath, so the tests will run even when compiling with different Scala versions.

The Scala compiler options for Scala 3 are largely incompatible with the Scala 2.13 ones, so I've set them to [] for now as I don't know what the preferences of the Kafka developers are regarding that.

[jlprat]

Thanks @mberndt123 for your PR

gradlewAll needs to be modified as well. But as I mentioned in the mailing list, I think it would be beneficial to start a "DISCUSS" thread with this proposal, as there are a couple of things we might want to align.

[jlprat]

What's the motivation for this change? This means that we would run scalafmt with 2.13 rules regardless of the Scala version we compile. I think we could use something like https://scalameta.org/scalafmt/docs/configuration.html#scala-2-with--xsource3.

[mberndt123]

It's a leftover from an old version.
Originally I using the same scalaVersion for both Kafka-Streams-Scala and Core, and there is no Scala 3 artifact for scalafmt yet, so it wasn't working.
But since now I'm using separate versions, I'll undo this change.

[jlprat]

This shouldn't be here I believe.

[mberndt123]

OK, I've removed it

[mberndt123]

Thanks @mberndt123 for your PR

gradlewAll needs to be modified as well. But as I mentioned in the mailing list, I think it would be beneficial to start a "DISCUSS" thread with this proposal, as there are a couple of things we might want to align.

The only reasonable change would be to add && ./gradlew "$@" -PscalaVersion=2.13 -PstreamsScalaVersion=3 to the script. But then when you run something like ./gradlewAll :core:publish, it's going to run that task twice for Scala 2.13, which is probably a bad idea. So I wouldn't know how to change that script in a useful way 🤷🏻‍♂️

[altrack]

@mberndt123 one of the possible solutions may be to attach the Scala 3 related step as a piggyback to Scala 2.13, it will be skipped when scalaVersion=2.12. This until we have a full Scala 3 release.
It can be done by adding an additional project called :streams:streams-scala3 which will be dependent on :streams:streams-scala and can be excluded by -x :streams:streams-scala3.
No change in gradlewAll or Jenkinsfile required.
So, by default and with scalaVersion=2.13 it will provide us with 2 artifacts for 2.13 and 3.
Also, in this case I don't see a benefit of having Scala 3 versions, as longs as those as LTS it should just work.

[dychko]

Any chance of getting this merged?
We also want to use kafka streams with Scala 3, so would appreciate it!