rhel9-ppc64le VMs offline due to NVMe failure at OSUOSL

[richardlau]

Got this email from OSUOSL:

One of our nodes which uses local storage with NVMe has had a failure and all the VMs on that node are offline. Due to the way we had storage configured on this node for performance, we're unlikely to save any of the ephemeral disks. However if you had any volumes attached, those will still be intact since they are on our Ceph cluster.

I'm going to be working on getting the NVMe replaced, in the meantime, if you need us to rebuild any of your VMs, please let us know.

This affects our rhel9-ppc64le VMs, which I had set up with NVMe storage as they had resulted in marginally faster builds:

• test-osuosl-rhel9-ppc64_le-1 is DOWN jenkins-alerts#3262
• test-osuosl-rhel9-ppc64_le-2 is DOWN jenkins-alerts#3263
• test-osuosl-rhel9-ppc64_le-3 is DOWN jenkins-alerts#3264

[mhdawson]

Removed rhel9-ppc64le from these jobs:

• https://ci.nodejs.org/job/node-test-commit-plinux/configure until we get the machines restored.

@richardlau said he'd take a look tomorrow.

[mhdawson]

There are a couple of other jobs that use rhel9-ppc64le, but they have not runs for 5-6 days so they are more infrequent so I've not disabled them yet.

I think we'll get the machine back tomorrow, if not we can exclude those as well:

• https://ci.nodejs.org/job/citgm-smoker/nodes=rhel9-ppc64le/
• https://ci.nodejs.org/job/citgm-smoker-nobuild/nodes=rhel9-ppc64le/

[richardlau]

No updates on the NVMe replacement. I'm going to create new (non-NVMe) VMs.

[richardlau]

I've created a new test-osuosl-rhel9-ppc64_le-4 and replaced test-osuosl-rhel9-ppc64_le-3 (Ansible inventory update in #3999) with non-NVMe RHEL 9 VMs. That will give us two machines for now.

My plan is to wait to see what the outlook is for getting the NVMe replaced before deciding what to do about test-osuosl-rhel9-ppc64_le-1 and test-osuosl-rhel9-ppc64_le-2.

[richardlau]

I've updated node-test-commit-plinux to restore rhel9-ppc64le.

CI runs:

• test-osuosl-rhel9-ppc64_le-3 - https://ci.nodejs.org/job/node-test-commit-plinux/57237/nodes=rhel9-ppc64le/
• test-osuosl-rhel9-ppc64_le-4 - https://ci.nodejs.org/job/node-test-commit-plinux/57236/nodes=rhel9-ppc64le/

[mhdawson]

@richardlau thanks for the quick work.

[richardlau]

The system is back online and you should be able to recreate the VMs now. I unfortunately needed to remove all current VMs to get the hypervisor back in a sane state so using "rebuild" will not work right now.

So

• test-osuosl-rhel9-ppc64_le-1
• test-osuosl-rhel9-ppc64_le-2

are gone. I'll recreate them and update #3999 with the new IP addresses (I'm assuming we'll get new IP addresses with new VMs).

[richardlau]

Recreated

• test-osuosl-rhel9-ppc64_le-1
• test-osuosl-rhel9-ppc64_le-2

as p9.nvme.large (Power 9, 4 VCPUs, 8 GB RAM, 80 GB disk).

For now, I'll keep the other two machines as non-NVMe as they are working. We can revisit later if we want to convert those to NVMe (would involve removing the existing VM and reprovisioning).

[richardlau]

which I had set up with NVMe storage as they had resulted in marginally faster builds

For posterity (and to put some data behind the "marginally faster builds" statement):

CI runs:

• test-osuosl-rhel9-ppc64_le-3 - https://ci.nodejs.org/job/node-test-commit-plinux/57237/nodes=rhel9-ppc64le/
• test-osuosl-rhel9-ppc64_le-4 - https://ci.nodejs.org/job/node-test-commit-plinux/57236/nodes=rhel9-ppc64le/

Without NVMe took 2 hours and 2 hours 9 mins (both without a populated ccache).

• test-osuosl-rhel9-ppc64_le-1 https://ci.nodejs.org/job/node-test-commit-plinux/57284/nodes=rhel9-ppc64le/
• test-osuosl-rhel9-ppc64_le-2 https://ci.nodejs.org/job/node-test-commit-plinux/57285/nodes=rhel9-ppc64le/

With NVMe took 1 hour 51 mins and 1 hour 45 mins (both without a populated ccache).