Move data8x home storage to our own NFS Server

[yuvipanda]

We currently use Google cloud filestore for user home directory storage on data8x. This is more expensive than we'd like, and @felder is happy to maintain an NFS Server similar to what we do for datahub. So for cost efficiency reasons, let's switch over.

This would require:

• Set up new NFS Server in some semi-automated fashion
• Decide on storage mechanisms, set up NFS server
• Mount current filestore to new VM, start rsync'ing files over
• Do iterative rsync's until rsync time is very small
• Announce multi-hour downtime window to users
• Turn off the hub and kill all user pods, so we can migrate the user home directories to new NFS server
• Decomm old filestore after a week

[felder]

Some notes regarding storage:

The storage mechanism is using standard google persistent disks formatted with an ext4 filesystem. Resizing google persistent disks upward along with ext4 is easy and quick without requiring unmounting and downtime. Best practices suggest using one persistent disk per filesystem. This ensures we can use Google's disk snapshotting tool to perform backups without risking inconsistencies that can occur when filesystems span multiple disks. Google persistent disks have a max capacity of 64TB which is well over the amount required. Furthermore moving Google persistent disks between VMs is trivial without having to worry about performing steps before and after the move. All of this makes using ZFS or LVM to span multiple disks unnecessary, more complex, and less desirable.

The main issue with this approach is resizing filesystems downward. Google persistent disks cannot be shrunk. However, most filesystems including ext4 cannot be shrunk while online. Thus while it is possible to imagine a scheme where LVM is used with multiple physical volumes to shrink available storage without needing to rsync, the subsequent filesystem resizing could not be done online. Thus we gain very little over provisioning new storage and rsyncing as was done for this issue.

[felder]

@chrispyles noted an issue today where some students were getting an error indicating an inability to create a home directory. This was the result of the permissions not being set correctly on the nfsserver. For some reason on the google filestore, the parent directory "home" for the student directories was owned by root and it somehow worked. I had to change ownership of the directory so it was owned by ubuntu. This is similar to how the permissions for the same directory are set on the other nfsserver mounts.

[yuvipanda]

I think this is done, so am gonna close this.