Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: Migrate to new network storage on MacStadium #33155

Closed
toothrot opened this issue Jul 17, 2019 · 7 comments

Comments

@toothrot
Copy link
Contributor

commented Jul 17, 2019

MacStadium is deprecating the old storage SAN for a new NFS based storage. We should be able to automate moving the data over in govc, then updating makemac to use the new vSphere datastore name.

@toothrot toothrot added this to the Unreleased milestone Jul 17, 2019

@toothrot toothrot self-assigned this Jul 17, 2019

@toothrot

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2019

The new NFS-based storage in the new MacStadium datacenter is now available. I've been copying the vm base images (osx_14_frozen, osx_12_frozen, and so on) to the new storage GGLGLN-A-001-STV1, adding a _nfs suffix: osx_14_frozen_nfs.

Once these are up, we should be able to make a change to makemac.go to reference the new image names and location.

We'll probably want to roll this out on a night or a weekend to test. We should be able to verify by manually starting a VM for each image with the same command that makemac uses.

The final step will be to move the bastion VM box as well. I haven't yet done enough research to determine the necessary steps there yet, but vSphere's snapshotting functionality seems solid.

@gopherbot

This comment has been minimized.

Copy link

commented Jul 30, 2019

Change https://golang.org/cl/188219 mentions this issue: cmd/makemac: use new MacStadium datastore

@toothrot

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2019

I was able to move the dns host / bastion / makemac vm to the new storage with vSphere's live migration tool.

I cloned all the base VMs to the new storage, and ensured that a snapshot ran on each one for use by cmd/makemac/makemac.go#CreateMac. I tested starting a VM, and verified that it successfully connected to the coordinator.

After https://golang.org/cl/188219, we should be able to restart makemac and see if the new VMs it creates (which it does regularly) work correctly.

Once we've restarted all VMs on the new storage, we can unmount the old datastore in vSphere and allow it to be decommissioned.

gopherbot pushed a commit to golang/build that referenced this issue Aug 6, 2019

cmd/makemac: use new MacStadium datastore
MacStadium is migrating off the current datastore. As part of the
transition, we copied our base images to the new storage, adding _nfs as
the suffix. The new images were snapshotted to allow for copy on write.

New base image names were suffixed with _nfs to accommodate the
migration. The new storage device was named GGLGLN-A-001-STV1 by
MacStadium.

Updated vmware-notes.txt to include manual instructions for a test run
of this latest change. The VM started and successfully connected to the
coordinator.

Updates golang/go#33155

Change-Id: I69976ebc6faa4f95a0aefd4d2e4c8d6a5bcf8a0f
Reviewed-on: https://go-review.googlesource.com/c/build/+/188219
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@toothrot

This comment has been minimized.

Copy link
Contributor Author

commented Aug 6, 2019

A new version of makemac has been deployed, using the new datastore. All running VMs are now on the new datastore.

I'm working on deprovisioning the old datastore, as our hosting provider has given us a surprise deadline of August 9th.

@toothrot

This comment has been minimized.

Copy link
Contributor Author

commented Aug 6, 2019

The old datastore (Pure1-1) is unmounted. Our hosting provider is working to remove it.

@andybons

This comment has been minimized.

Copy link
Member

commented Aug 23, 2019

Has this work been completed?

@toothrot

This comment has been minimized.

Copy link
Contributor Author

commented Aug 23, 2019

Yes!

The migration has been completed, and the old storage has been decommissioned. There are still some references to it in vSphere that I cannot figure out how to get rid of, but it does not seem to be causing any issue.

@toothrot toothrot closed this Aug 23, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.