adding the patch into H2O 3 code to re-connect to the h2o nodes

[exalate-issue-sync]

customer was facing the issue of cluster disconnecting and 401 authentication error
work around was using the script below and customer would like to add as patch fix for coming up h2o-3 version:

def _query_job_status_safe(self):
import time
from h2o.exceptions import H2OConnectionError, H2OResponseError, H2OServerError
result = None
attempts = 0
last_err = None
while attempts < 300:
try:
attempts += 1
result = h2o.api("GET /3/Jobs/%s" % self.job_key)
self.job_poll_success = True # only retry if there was at least one OK response
break
except (H2OConnectionError, H2OResponseError, H2OServerError) as e:
last_err = e
if self.job_poll_success:
print("Job request failed %s, will retry after 3s." % e.args[0])
time.sleep(3)
else:
raise e
if result:
return result
else:
raise last_errdef _refresh_job_status(self):
if self._poll_count <= 0: raise StopIteration("")
jobs = self._query_job_status_safe()
self.job = jobs["jobs"][0] if "jobs" in jobs else jobs["job"][0]
self.status = self.job["status"]
self.progress = self.job["progress"]
self.exception = self.job["exception"]
self.warnings = self.job["warnings"] if "warnings" in self.job else None
self._poll_count -= 1

Sometimes the server may report the job at 100% but still having status "RUNNING" – we work around this

by showing progress at 99% instead. Sometimes the server may report the job at 0% but having status "DONE",

in this case we set the progress to 100% manually.

from h2o.utils.shared_utils import clamp
if self.status == "CREATED": self.progress = 0
if self.status == "RUNNING": self.progress = clamp(self.progress, 0, 0.99)
if self.status == "DONE": self.progress = 1
if self.status == "FAILED": raise StopIteration("failed")
if self.status == "CANCELLED": raise StopIteration("cancelled by the server")
return self.progressfrom h2o.job import H2OJob

H2OJob._refresh_job_status = _refresh_job_status
H2OJob._query_job_status_safe = _query_job_status_safe

suports:
[https://support.h2o.ai/a/tickets/98708|https://support.h2o.ai/a/tickets/98708]
[https://support.h2o.ai/a/tickets/98776|https://support.h2o.ai/a/tickets/98776]

[h2o-ops]

JIRA Issue Details

Jira Issue: PUBDEV-8034
Assignee: Michal Kurka
Reporter: Retima Dangol
State: Resolved
Fix Version: 3.32.0.5
Attachments: N/A
Development PRs: Available

[h2o-ops]

Linked PRs from JIRA

#5448
#5365
#5366