Kubernetes API lockdown after clustering

[exalate-issue-sync]

When H2O is deployed in Kubernetes environment, only the leader node should be exposed. This is currently done via a mechanism where H2O makes itself signal "unready" state to the K8S service, except for the leader node. Yet the client APIs on HTTP protocol still work on each node, making it easy to be mistakenly called by the client - which may happen in cases where H2O is somehow wrongly exposed in K8S environment (e.g. wrong type of service used).

Possible solutions are:

Disable client API of all nodes except for leader node once the clustering is done.

Throw 4xx HTTP error, e.g. 403 - Forbidden with a meaningful error message informing the user about using wrongly set-up cluster.

Other objectives are:

Investigate to what extent can H2O diagnose the K8S setup it is spawned in - mainly the type of service used. This could potentially be a problem with roles and priviliges configuration and may not be enabled on all K8S implementations. What we could do is to at least attempt to perform such checks (provided they make sense) and warn the user about not being able to perform them - this does not prevent the clustering to happen.

Consult with Sparkling Water [~accountid:5c9943ec3a5542225fedb6b9] if a flag to enable the APIs is required. Or could be implemented otherwise - the APIs are only shut down when the flag is present, which changes the behavior of the above-described.

[h2o-ops]

JIRA Issue Migration Info

Jira Issue: PUBDEV-7753
Assignee: Pavel Pscheidl
Reporter: Pavel Pscheidl
State: Resolved
Fix Version: 3.32.0.3
Attachments: N/A
Development PRs: Available

Linked PRs from JIRA

#4917
#5134
h2oai/sparkling-water#2397