Should we use this project current?

[sola1tmy]

We are consider integration some spark related functions in our project.
But seems the progress for support spark 3.4 in this project still have a lot of work.

#195

we consider

• java: it has a poor grammer to do hard work, but maybe the best way now, because it can communicate with both kotlin and spark
• scala: seems hard communicate with kotlin since our main project is write in java and kt. Maybe there will have problems hard to solve.
• kotlin: That's what I most like. But unfortunely Spark 3.4+ / -Connect support #195 remains, we can still use old version for a quite long time, but migration is Inevitable maybe after 1 or 2 years.

So should we use this project now? when will the next release come?And will the api keep same next release?

[Jolanrensen]

Hi!
Thanks for your interest :) I'd say, if you can use the latest version of Spark (3.3) supported by the Kotlin API in your project, you could at least try if it works for you. The plan for 3.4+ is to keep the migration as small as possible.

I'm not yet sure whether the Kotlin Spark API should move to spark-connect only or also generic spark, but if you keep your project to the concepts mentioned here you should be good to go for any direction.
The main thing to take away is to try to stick to your own defined data classes for Datasets (so try to avoid Pair, Triple, and data classes from other libraries). This will make the eventual transition easier as for 3.4, encodable Kotlin data classes will have to be annotation with @Sparkify.