H2OContext can't be initalized on Databricks cloud

[exalate-issue-sync]

In databricks environment, create cluster, add library - sparkling-water-assembly jar and create notebook with

{code:java}
val hc = H2OContext.getOrCreate(spark).
{code}

This will fail on class not found exception: org.spark_project.jetty.servlet...

The reason for this is that Spark normally uses shaded jetty in package org.spark_project.jetty and we use it as well

sparkling-water/core/src/main/scala/org/apache/spark/h2o/ui/SparklingWaterInfoPage.scala

Line 29 in 81d36ce

import org.spark_project.jetty.servlet.{DefaultServlet, ServletContextHandler, ServletHolder}

However Databricks does not use this shaded jetty an uses regular jetty, that is why we see this exception.

The first idea won't work - change shaded jetty to regular jetty in the class above as we are relocating H2O jetty ot a new package and this class would be relocated as well. We might try to forbid relocation for this particular class as here, we need to use just spark jetty.

The current workaround is to disable SParkling Water UI Tab as

{code:java}
val conf = new H2OConf(spark).set("spark.ui.enabled", "false")

val hc = H2OContext.getOrCreate(spark, conf)
{code}

[DinukaH2O]

JIRA Issue Migration Info

Jira Issue: SW-683
Assignee: Jakub Hava
Reporter: Jakub Hava
State: Resolved
Fix Version: 2.0.23
Attachments: N/A
Development PRs: Available

Linked PRs from JIRA

#544

[hasithjp]

JIRA Issue Migration Info Cont'd

Jira Issue Created Date: 2018-01-19T02:43:30.880-0800