Move off manually maintained NFS servers on GCP

[yuvipanda]

Context

On pilot-hubs cluster as well as cloudbank (I think?) user home directories are on a hand-rolled NFS VM. This is problematic - when it fills up, it is a bit complex to resize, and while resizing a PD is fairly standard, I don't think we have it documented.

We should just switch over to paying money for Google Filestore.

This is one of those issues that isn't an issue until it is, and when it is an issue, it's a big issue.

Next steps

• Figure out which clusters are using hand-rolled NFS
• Modify appropriate cluster tfvars to enable google filestore creation
• Migrate data from current NFS server to google filestore
• Decomission old NFS VM

[choldgraf]

@yuvipanda could you provide a basic explanation of the steps we'd need to take, or how complicated it would be? Anything that could help us understand when to prioritize this and how much work it would be.

[yuvipanda]

Only the shared 2i2c cluster on GCP is still using a manual NFS server now. Let's get rid of it before it goes critical and requires someone with XFS knowledge to fix.

[yuvipanda]

I did a copy of the files from the old NFS server to filestore by:

1. Mounting the filestore on the existing NFS server, with sudo mount -t nfs -o soft,noatime 10.234.45.250:/homes/ filestore/
2. Copying the existing files over, with time sudo cp -rav /export/home-01/homes/ /export/filestore/. This took about 31h to complete.

I'm going to run an rsync (time sudo rsync -a -P --delete /export/home-01/homes/ /export/filestore/homes/) now, and then I'll move over hubs one by one based on when they are unused.

[yuvipanda]

rsync takes about 44 minutes now.