[SW-1501] Check the case of JAR being attached prior the start of PySparkling #1433
Conversation
| here = path.abspath(path.dirname(__file__)) | ||
| if '.zip' in here: | ||
| with zipfile.ZipFile(here[:-len("ai/h2o/sparkling/")], 'r') as archive: | ||
| return archive.read('ai/h2o/sparkling/version.txt').decode('utf-8').strip() |
There was a problem hiding this comment.
Can happen that the version file is still in the pysparkling folder?
There was a problem hiding this comment.
Good question, but not anymore, we are generating it into ai/h2o/sparkling. As the ai/h2o/sparkling is our core backend python library & Implementation if initializer is there, I thought it makes sense to have the version file there as well
|
@mn-mikke we are not changing the order of dependencies. Spark is still loading the dependencies the same way as before. What we just changed is that we added verification of the Sparkling Water JAR. We check if the JAR is already attached to the cluster and if not, attach it. We have already been attaching the JAR before, but we ignored the fact that the JAR might have been already attached. What will change for the user that they will see exception when they have conflicting jars, but that is a bug fix as before, they could hit weird issues. This way, we warn them early |
No description provided.