Mac Stadium outage for upgrade Dec 16, 9-11 ET

[mhdawson]

An orka upgrade is required for the infra that supports some of our MacOS machines.

This time seemed to be the best of what was offered as I did not think waiting until the week before the holidays made sense.

Info related to the upgrade from MacStadium

In order to complete your upgrade as smoothly as possible, please review the following:

We require a minimum 2 hour maintenance window to perform the upgrade. We have time slots (EST) to chose from Monday-Thursday.
The API will be unreachable during the maintenance window. Please pause all CI/CD functions during the maintenance window.
Prior to starting the maintenance window, any running VMs need to be saved by you and your team (orka image save or orka image commit).
For upgrades to 1.7, all VMs will need to be shutdown from the cluster and drained prior to the start maintenance window.
There is no impact to your images, VM configs or ISOs performing this upgrade.
We will notify you as soon as the upgrade is completed!

@AshCripps any chance you will be around in terms of helping out with the pre/post activities?

[AshCripps]

Yep I should be available to help out.

[mhdawson]

@AshCripps thanks, we should try to get one of use (Me, Richard or sxa) to watch while you do the steps as some knowledge transfer.

[rvagg]

Expect some possible downtime this week:

We can wait to perform the full upgrade until next Thursday, however, we can temporarily fix the cert issue by applying a patch that should have minimal impact and prevent an outage this week. We would need your permission to execute this. Please let us know.

I've just told them to go ahead and do what they need and we'll manage on our end.

[mhdawson]

@rvagg thanks for the update. I wonder if we can swap in the build email so that we all get the updates?

[rvagg]

We have patched the certificate issue and the cluster should be good. We still plan on upgrading the cluster and expanding it next Thursday during the scheduled window. Let us know if you have any questions prior to then.

Yeah, I'd like to swap the emails on this account, not sure how though. Will try and figure it out this week.

[mhdawson]

@AshCripps the interval starts at 9ET so I think the shutdown steps will be needed beforehand. Maybe you can do that and then you can help us walk through bringing things back up after the upgrade around 11ET? If that makes sense I'll send out an invite.

[rvagg]

FYI I've added @mhdawson @richardlau @sxa to our MacStadium account. I'm not sure you'll get emails but Michael managed to log in somehow so it works. You should all get the emails that I've been getting each time a ticket has action in their system.

[richardlau]

I can confirm that I got an email from MacStadium and was able to log in.

[AshCripps]

@mhdawson is there a list of preshutdown steps they want done? im not in any email communication.

[mhdawson]

This is what was in the notice (from top comment in this issue)

Prior to starting the maintenance window, any running VMs need to be saved by you and your team (orka image save or orka image commit).
For upgrades to 1.7, all VMs will need to be shutdown from the cluster and drained prior to the start maintenance window.

I think we should assume we are having an upgrade to 1.7 to be safe?

[mhdawson]

@AshCripps

[AshCripps]

ok I think its probs worth blowing the vms away and starting afresh anyway as some are in a weird state of brokeness - there are base images already though so should be ok.