Tried using port 54321 for Flow proxy, but port was already occupied

[exalate-issue-sync]

I've a Spark application with dependencies on H2O Sparkling Water jars

// H2O Sparkling Water version (spark 2.4)
val h2oVersion = "3.32.0.1-2-2.4"
lazy val h2oLibs = Seq(
  "org.apache.spark" %% "spark-repl" % sparkVersion,
  "ai.h2o" % "sparkling-water-package_2.11" % h2oVersion //exclude("ai.h2o", "sparkling-water-api-generation")
)

When I try to start it on k8s, the Spark executor starts properly (see logs from executor container below) and I can open a port-forward connection to it and access flow UI.
But the driver node gets stack in infinity loop and I see in the logs the following

2020-12-03T17:04:59.532+0000 level=INFO thread=main logger=ai.h2o.sparkling.backend.utils.RestApiUtils
H2O node http://10.1.0.22:54321/3/verifyWebOpen successfully responded for the GET.
2020-12-03T17:04:59.553+0000 level=INFO thread=main logger=ai.h2o.sparkling.backend.utils.RestApiUtils
H2O node http://10.1.0.22:54321/3/verifyVersion?referenced_version=3.32.0.1 successfully responded for the GET.
2020-12-03T17:04:59.636+0000 level=INFO thread=main logger=ai.h2o.sparkling.backend.utils.RestApiUtils
H2O node http://10.1.0.22:54321/3/Cloud successfully responded for the GET.
2020-12-03T17:04:59.713+0000 level=INFO thread=main logger=ai.h2o.sparkling.backend.utils.RestApiUtils
H2O node http://10.1.0.22:54321/3/LogLevel successfully responded for the GET.
2020-12-03T17:04:59.951+0000 level=INFO thread=main logger=ai.h2o.sparkling.backend.utils.RestApiUtils
H2O node http://10.1.0.22:54321/99/Rapids successfully responded for the POST.
2020-12-03T17:05:00.011+0000 level=WARN thread=main logger=ai.h2o.sparkling.backend.utils.ProxyStarter
Tried using port 54321 for Flow proxy, but port was already occupied!

Apparently, when SW tries to check for available ports it uses the Spark driver host but later jetty server binds to 0.0.0.0

See here for detailed discussion #2414

[DinukaH2O]

JIRA Issue Migration Info

Jira Issue: SW-2503
Assignee: dzlab
Reporter: dzlab
State: Resolved
Fix Version: 3.32.0.3-1
Attachments: N/A
Development PRs: Available

Linked PRs from JIRA

#2425

[hasithjp]

JIRA Issue Migration Info Cont'd

Jira Issue Created Date: 2020-12-15T21:58:53.053-0800