Support combining SSDs into a raid

[csweichel]

Useful for GKE deployments

[mrsimonemms]

/lgtm

[roboquat]

LGTM label has been added.

Git tree hash: cdd2fb617237261659421cd5d2f86682773fd08f

[csweichel]

/approve no-issue

[roboquat]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csweichel, MrSimonEmms

Associated issue requirement bypassed by: csweichel

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [MrSimonEmms,csweichel]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jankeromnes]

@csweichel @aledbf @mrsimonemms Hi!

FYI, this change seems to be related to a cluster incident where a ws-daemon pod keeps crashlooping (happened several times since this PR was deployed).

The symptoms are:

• A ws-daemon pod keeps crashlooping, triggering an alert

• kubectl describe shows this event:

$ kubectl describe pod ws-daemon-zc64j
[...]
Events:
  Type     Reason   Age                   From     Message
  ----     ------   ----                  ----     -------
  Warning  BackOff  112s (x390 over 86m)  kubelet  Back-off restarting failed container

• When getting the logs for all containers, it seems to be the first "init container", raid-local-disks, that fails with:

...
+ DISK_DEV_SUFFIX=(c d e f g h i j)
+ MAX_NUM_DISKS=8
+ NUM_DISKS=0
+ declare -a DISKS
++ seq 0 7
+ for i in `seq 0 $((MAX_NUM_DISKS-1))`
+ CURR_DISK=/dev/sdc
+ '[' '!' -b /dev/sdc ']'
+ break
+ '[' 0 -eq 0 ']'
+ echo 'no local disks detected!'
+ exit 1
no local disks detected!

[jankeromnes]

Update: We now believe the code is behaving as expected (after also #5096), i.e. "fail to init ws-daemon if setupSSDRaid is enabled but there are no disks for it".

The incident looks rather like a configuration problem (i.e. there is a headless pool in EU, which is unexpected, and it seems to both enable setupSSDRaid and have no disks for it, leading its single ws-daemon to crashloop).