registry-facade: Ensure that node-labeler always monitors the registr-facade container

[utam0k]

Description

When only the container of the registry-facade was broken, the ready-nobel of the node was not disappearing because when the registry-facade container restarts, other containers even in the same pod don't restart.
Kubernetes docs about that

This PR monitors the state of the registry-facade with the node-labeler's liveness probe. If the registry-facade container is not ready, the prestop hooks of node-labeler will remove the label.

Previously, I thought to address this by changing the ready-probe-labeler. However, we realized that it would be simpler to go with the Kubernetes mechanism.
#15021

Related Issue(s)

Fixes: #13915

How to test

Please check this video more detail
https://www.loom.com/share/ed6f97ecaa9a4c249309f847af522308

• Node label, gitpod.io/registry-facade_ready_ns_default disappears when deleting a registry-facade container and turn on when registry-facade come back
• Node label, gitpod.io/registry-facade_ready_ns_default disappears when deleting a registry-facade pod and turn on when registry-facade come back

Release Notes

Do not land workspaces on the node with broken registry-facade

Documentation

Werft options:

• /werft with-local-preview
  If enabled this will build install/preview
• /werft with-preview
• /werft with-large-vm
• /werft with-integration-tests=all
  Valid options are all, workspace, webapp, ide, jetbrains, vscode, ssh

[utam0k]

@aledbf What do you think about this approach?

[kylos101]

@utam0k may I ask, could you also consider adding a similar probe to node-labeler container in ws-daemon daemonset? You'll see here, that it would benefit from the same pattern.

[kylos101]

This makes sense to me, nice work, @utam0k ! In other words, this liveness probe will only prevent scheduling of future workspaces (until the label is added back), it won't interrupt workspaces that are actively pulling images on start.

[kylos101]

@utam0k why are node-labeler PeriodSeconds time 2, and registry-facade 10?

The side effect is we'll be more likely (when network conditions are poor) to remove the node label, but not necessarily restart the registry-facade container (until more time has passed).

I assume intentional?

I am going to include @sagor999 , so that he can check if this will have an unintended consequence, and whether that is acceptable or not.

For example, if when we remove the label, that may cause the autoscaler to provision a new node, because more Re: workspace pods are unscheduable (ultimately, probably a good thing).

[jenting]

LGTM

/hold
waits for @sagor999 review

[sagor999]

/unhold