Support for Spark 2.4

[jeynesrya]

Is there a plan to add support for Apache Spark 2.4?

[vglagoleva]

@bsikander @Nibooor and I have it on our TODO list. But we won't probably contribute it soon.

If you are interested in the topic, contributions are very welcome :)

[murraytodd]

Are there any known issues that need to be addressed, or is this simply a matter of updating various dependency versions and testing? If the latter, I may be able to contribute time & effort.

[bsikander]

@murraytodd actually I have already implemented 2.4 upgrade. It will be pushed soon. Stay tuned :)

[murraytodd]

@murraytodd actually I have already implemented 2.4 upgrade. It will be pushed soon. Stay tuned :)

Brilliant! I'll relay that to my team!

[jeynesrya]

@murraytodd actually I have already implemented 2.4 upgrade. It will be pushed soon. Stay tuned :)

That's brilliant news! That release will be a game changer for us - thanks!

[bsikander]

@murraytodd @jeynesrya great that it helps you but please note that k8s support is still not there.

[murraytodd]

@murraytodd @jeynesrya great that it helps you but please note that k8s support is still not there.

That's fine. It's the improved Spark Structured Streaming in 2.4 that we've been waiting for. (And being able to support general Spark 2.4 environments. SJS is our application's bridge (and is very important based on some of our just-in-time computational functionality) so it dictates our Spark version dependencies.

[bsikander]

@jeynesrya @murraytodd 2.4.4 change is merged in master. Next release of jobserver (coming soon) will also support 2.4.4.

[jeynesrya]

That's excellent news. Will the release include a docker image pushed in Dockerhub or is that for each person to do individually?

[bsikander]

Regarding automatic publishing of docker, i am not sure. I will get back to you.

[bsikander]

@jeynesrya currently, we have not pusblished to docker. The machinery is not ready. For now, you have to go indivually. In future, we plan to add this to the automatic publishing.

Feel free to contribute here.

[jeynesrya]

@jeynesrya currently, we have not pusblished to docker. The machinery is not ready. For now, you have to go indivually. In future, we plan to add this to the automatic publishing.

Feel free to contribute here.

Thanks for the update! If I manage to get the job server running in Docker, I’ll contribute as mentioned 😊

[bsikander]

You can find an example here. I did this some time ago but it needs to be updated and improved.
https://hub.docker.com/repository/docker/bsikander/sjs

[bsikander]

Closing issue since jobserver has been release with this version.