Unable to create h2o_context on Databricks using R and Scala #1193
Comments
|
Hi,
Currently, internal mode of Sparkling Water does not support auto-scaling. It’s on our long-term roadmap, however for now, please turn off auto-scaling and dynamic allocation.
Thanks,
Kuba
[https://www.dropbox.com/s/rzkinjajjoexlcn/signlogo.png?raw=1]
Jakub Háva
Senior Software Engineer
jakub.hava@h2o.ai<mailto:jakub.hava@h2o.ai>
H2O.ai, The Open Source Leader in AI and ML<http://www.h2o.ai/>
On May 12, 2019, at 11:55 AM, Sasi Kiran Malladi <notifications@github.com<mailto:notifications@github.com>> wrote:
I'm trying to use sparkling water on Azure Databricks and I'm not able to create h2o_context. I tried this in both R and Scala on the same cluster.
R Sample code:
install.packages("sparklyr")
install.packages("rsparkling")
install.packages("h2o", type="source", repos="https://h2o-release.s3.amazonaws.com/h2o/rel-yates/2/R")
library(rsparkling)
library(sparklyr)
sc <- spark_connect(method="databricks")
h2o_context(sc)
Scala sample code:
import org.apache.spark.h2o._
val hc = H2OContext.getOrCreate(spark)
Configuration details
* Azure Databricks version: 5.3 ML (includes Apache Spark 2.4.0, Scala 2.11)
* Driver type: Standard_DS3_V2: 14.0 GB Memory, 4 Cores, 0.75 DBU
* Min workers: 2
* Max workers: 8
* Enable autoscaling: Yes
* Sparkling Water library: sparkling_water_assembly_2_11_2_4_10_all.jar
Error log:
Error : org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 14.0 failed 4 times, most recent failure: Lost task 1.3 in stage 14.0 (TID 144, 10.139.64.6, executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages. Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:2355) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:2343) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:2342) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2342) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:1096) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:1096) at scala.Option.foreach(Option.scala:257) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1096) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2574) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2522) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2510) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:893) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2233) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2255) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2274) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2299) at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:961) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:379) at org.apache.spark.rdd.RDD.collect(RDD.scala:960) at org.apache.spark.h2o.backends.internal.InternalBackendUtils$class.startH2O(InternalBackendUtils.scala:196) at org.apache.spark.h2o.backends.internal.InternalBackendUtils$.startH2O(InternalBackendUtils.scala:306) at org.apache.spark.h2o.backends.internal.InternalH2OBackend.init(InternalH2OBackend.scala:104) at org.apache.spark.h2o.H2OContext.init(H2OContext.scala:129) at org.apache.spark.h2o.H2OContext$.getOrCreate(H2OContext.scala:403) at org.apache.spark.h2o.H2OContext$.getOrCreate(H2OContext.scala:438) at org.apache.spark.h2o.H2OContext.getOrCreate(H2OContext.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sparklyr.Invoke.invoke(invoke.scala:139) at sparklyr.StreamHandler.handleMethodCall(stream.scala:123) at sparklyr.StreamHandler.read(stream.scala:66) at sparklyr.BackendHandler.channelRead0(handler.scala:51) at sparklyr.BackendHandler.channelRead0(handler.scala:4) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<#1193>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABBMAS5CXDH3FUYJZERISOLPU7SQPANCNFSM4HMJ74JQ>.
|
|
Hi @jakubhava, I created a new Databricks cluster without autoscaling and by setting number of workers to 4. I ran the same code in R and Scala and the same error happened. Databricks configuration: Error log: |
|
Can you please share yarn logs? |
|
I am running this on Databricks. There is no yarn integration. |
|
I believe this is not true. Internally, Databricks run their Spark on Hadoop as well. What we need is full logs from executors and driver |
|
I found these logs under spark driver logs. Are these sufficient? |
|
Thank you @sasikiran, will have a look |
|
Could you please try the nightly build from this link? We changed some internal configuration which my influence this behavior https://h2o-release.s3.amazonaws.com/sparkling-water/rel-2.4/nightly/91/index.html Thank you! |
|
Thanks @jakubhava. I used the nightly build on the same cluster with same code. The error remains same as well. Do you want me to share the logs again? |
|
Oki, thank you! I think it is fine, we will think about what we can do. |
|
Hi @sasikiran, we have been working on a better solution. Can you please give it another try and try this nightly release? This one should do http://h2o-release.s3.amazonaws.com/sparkling-water/rel-2.4/nightly/98/index.html |
|
Hi @jakubhava, thanks for the update. I tried it out using the new nightly build. I am getting a different error this time. I tried it in both R and Scala.
Here are the detailed logs: |
|
Thank you this is helpful! This is still fairly new change so we need to iterate. However, the main error happens on the executor machines. Could you please sent us logs from the executors? |
|
I was able to find these. Please let me know if these are good enough. Executor 1 log: https://file.io/NZsDMH |
|
Thanks, I had a look inside, but did not see useful logs. I will spent some time on it and try to reproduce myself. |
|
I created testing environment with the same configuration in Databricks Azure you have but still can't reproduce, testing using Scala with the simple code bellow: Quick question: Are you specifying any additional spark options? What the Error above says is that the the driver did not get response from H2O worker node, so the H2O worker node probably didn't start. The logs you sent do not contain information about this behavior. But you should be able to get the right logs of failing case when you go to Spark UI -> click on executors and send here stdout & stderr from the executors. That would help us a lot |
|
These screenshots should help with getting the right logs |
|
Hi @jakubhava, I think there is some conflict between databricks runtime and sparkling water. I created a new cluster and selected databricks runtime: As a workaround, I will use sparkling water on databricks runtime 5.3. Thanks so much! |
|
Hi @sasikiran thanks so much, this information will help us with debugging! |
|
@sasikiran just FYI: I was able to reproduce the issue Core issue is: |
|
That’s awesome.
… |
|
Hi, @jakubhava do you have any workarounds or recommendations on this issue? |
|
Adding @honzasterba as I discussed that with him. Do you please remember the issue with XGBoost conflicts we discussed on Slack? |
|
I think this was caused by there being two xgboost jars on the classpath - one from h2o and one from another library, this caused various issues with xgboost classes being present twice. |
|
Yup, exactly. Do you have any workaround on your mind? |
|
|
I see, thank you. One more question, which versions of spark and scala where used for sparkling water latest release? It seems, that I've found the issue with xgboost and fixed it, but now I've faced some issues with compiler. I'm using spark 2.4.3 with scala 2.11.12 on win10 x64 with hadoop 2.7.1 via winutils. Should I create a new issue on this problem?
|
|
Yes, please create a new issue for this one, thanks |
|
@MrHadgehog How did you fix XGBoost issue? I'm having same issue in my project. I've following XGBoost dependency in my project along with H2O Sparkling water dependency. ml.dmlc xgboost4j-spark 0.82 ai.h2o sparkling-water-package_2.11 3.28.0.1-1-2.3 |
|
@jakubhava Is there any way I can avoid this XGboost : java.lang.IllegalAccessError: tried to access class ml.dmlc.xgboost4j.java.NativeLibLoader from class hex.tree.xgboost.XGBoostExtension error? |
Hi @BhushG, sorry for the late reply. I removed dependencies ml.dmlc.xgboost4j or any similar from the entire project. After that the issue was solved. |
|
@MrHadgehog @jakubhava Hi, I found a turnaround for this problem. Apparently, the sequence of the dependencies matters in maven. So I moved the dependency of h2o above dependency of xgboost4j-spark and it worked. Thus h2o doesn't throw any error related to XGBoost |
|
hi, how to resolve this xgboost conflict error in azure databricks? |
|
Py4JJavaError: An error occurred while calling o411.getOrCreate. |
|
@garimagupta25 Can you use a plain non-ML version of Databricks runtime? |
|
Hello, bumping this issue again. I'm working in a databricks cluster in which I have XGBoost4j libraries installed. I'm trying to create a dummy model using Pysparkling water, finding the same issues described above. I cannot uninstall the xgboost libraries, since they're needed for other apps running on our cluster, and I cannot switch the cluster mode either. Thus, the solution proposed by @honzasterba is the only option to use Pysparkling for my team in our current setup:
Is it feasible to do it? I'm currently using pysparkling-water version 3.1 and the issue still remains. |
I'm trying to use sparkling water on Azure Databricks and I'm not able to create h2o_context. I tried this in both R and Scala on the same cluster.
R Sample code:
Scala sample code:
Configuration details
Error log:
Error : org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 14.0 failed 4 times, most recent failure: Lost task 1.3 in stage 14.0 (TID 144, 10.139.64.6, executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages. Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:2355) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:2343) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:2342) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2342) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:1096) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:1096) at scala.Option.foreach(Option.scala:257) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1096) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2574) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2522) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2510) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:893) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2233) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2255) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2274) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2299) at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:961) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:379) at org.apache.spark.rdd.RDD.collect(RDD.scala:960) at org.apache.spark.h2o.backends.internal.InternalBackendUtils$class.startH2O(InternalBackendUtils.scala:196) at org.apache.spark.h2o.backends.internal.InternalBackendUtils$.startH2O(InternalBackendUtils.scala:306) at org.apache.spark.h2o.backends.internal.InternalH2OBackend.init(InternalH2OBackend.scala:104) at org.apache.spark.h2o.H2OContext.init(H2OContext.scala:129) at org.apache.spark.h2o.H2OContext$.getOrCreate(H2OContext.scala:403) at org.apache.spark.h2o.H2OContext$.getOrCreate(H2OContext.scala:438) at org.apache.spark.h2o.H2OContext.getOrCreate(H2OContext.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sparklyr.Invoke.invoke(invoke.scala:139) at sparklyr.StreamHandler.handleMethodCall(stream.scala:123) at sparklyr.StreamHandler.read(stream.scala:66) at sparklyr.BackendHandler.channelRead0(handler.scala:51) at sparklyr.BackendHandler.channelRead0(handler.scala:4) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748)The text was updated successfully, but these errors were encountered: