importing pysparkling in Zeppelin fails on SW for 3.26.3

[exalate-issue-sync]

---

Py4JError Traceback (most recent call last)
in ()
1 import os.path
----> 2 from pysparkling import *
3 import h2o
4 #sc.setCheckpointDir('/user/' + user_name + '/checkpoints3/')
5

/opt/cloudera/parcels/Anaconda3/lib/python3.6/site-packages/pysparkling/init.py in ()
49 all = ["H2OContext", "H2OConf"]
50
---> 51 Initializer.load_sparkling_jar()

/opt/cloudera/parcels/Anaconda3/lib/python3.6/site-packages/ai/h2o/sparkling/Initializer.py in load_sparkling_jar()
47 if stream is None:
48 sys.path.append(".")
---> 49 Initializer.__add_sparkling_jar_to_spark(sc)
50 else:
51 otherVersion = jvm.scala.io.Source.fromInputStream(stream, "UTF-8").mkString()

/opt/cloudera/parcels/Anaconda3/lib/python3.6/site-packages/ai/h2o/sparkling/Initializer.py in __add_sparkling_jar_to_spark(sc)
71 url = jvm.java.net.URL("file://{0}".format(sw_jar_file))
72
---> 73 Initializer.__add_url_to_classloader(jvm, url)
74
75 # Add Sparkling Water Assembly JAR to Spark's file server so executors can fetch it

/opt/cloudera/parcels/Anaconda3/lib/python3.6/site-packages/ai/h2o/sparkling/Initializer.py in __add_url_to_classloader(jvm, url)
145 methods = [m.getName() for m in cl.getClass().getDeclaredMethods()]
146 if "addURL" in methods:
--> 147 cl.addURL(url)
148 logger.debug("Adding {} to classloader '{}'".format(url.toString(), cl.toString()))
149 else:

/opt/cloudera/parcels/SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in call(self, *args)
1255 answer = self.gateway_client.send_command(command)
1256 return_value = get_return_value(
-> 1257 answer, self.gateway_client, self.target_id, self.name)
1258
1259 for temp_arg in temp_args:

/opt/cloudera/parcels/SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179/lib/spark2/python/lib/pyspark.zip/pyspark/sql/utils.py in deco(*a, **kw)
61 def deco(*a, **kw):
62 try:
---> 63 return f(*a, **kw)
64 except py4j.protocol.Py4JJavaError as e:
65 s = e.java_exception.toString()

/opt/cloudera/parcels/SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
330 raise Py4JError(
331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
--> 332 format(target_id, ".", name, value))
333 else:
334 raise Py4JError(

Py4JError: An error occurred while calling o129.addURL. Trace:
org.apache.zeppelin.py4j.Py4JException: Method addURL([class java.net.URL]) does not exist
at org.apache.zeppelin.py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:335)
at org.apache.zeppelin.py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:344)
at org.apache.zeppelin.py4j.Gateway.invoke(Gateway.java:279)
at org.apache.zeppelin.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at org.apache.zeppelin.py4j.commands.CallCommand.execute(CallCommand.java:79)
at org.apache.zeppelin.py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Thread.java:748)

[exalate-issue-sync]

Jakub Hava commented: We had similar issue before, seems like regression in the last few releases.

[https://0xdata.atlassian.net/browse/SW-357?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel|https://0xdata.atlassian.net/browse/SW-357?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel|smart-link]

[exalate-issue-sync]

Jakub Hava commented: Will provide PR soon

[exalate-issue-sync]

Ruslan Dautkhanov commented: Thank you for quick turnaround Kuba !

[exalate-issue-sync]

Ruslan Dautkhanov commented: Is this possible to release SW 3.26.4 sooner with this fix, so we don’t have to wait for full H2O release cycle etc.

We can’t upgrade to 3.26 without this patch.

Thank you.

[exalate-issue-sync]

Jakub Hava commented: There are nightly releases which you can use and give it a try, however I would not suggest using them in production. I can trigger the nightly build tomorrow and send the link to try it out

Otherwise Sparkling Water releases are tighed to H2O relaases and SW 3.26.4 will be released after release of H2O 3.26.0.4

[exalate-issue-sync]

Ruslan Dautkhanov commented: What about 3.26.1.3 release? :)

This is part of Enterprise support to be able to request off-release builds.
https://www.h2o.ai/enterprise-support/

I often have to request patched CDH builds from Cloudera Support, for example.

This is a critical issue for us that prevents 3.26 upgrade..

I will send an email with more details.

Thank you.

[exalate-issue-sync]

Ruslan Dautkhanov commented: As you noticed we can't use nightly build in production

We'd like to have a build of 3.26.0.3 with this single patch.

Hope this is not too much trouble for you .

Again, I will send an email with more details.

Thank you

[DinukaH2O]

JIRA Issue Migration Info

Jira Issue: SW-1580
Assignee: Jakub Hava
Reporter: Nidhi Mehta
State: Resolved
Fix Version: 3.26.5
Attachments: N/A
Development PRs: Available

Linked PRs from JIRA

#1503

[hasithjp]

JIRA Issue Migration Info Cont'd

Jira Issue Created Date: 2019-08-29T11:30:53.402-0700