Blog post: Pangeo hub deployment

[choldgraf]

Description

We've recently wrapped up the initial deployment of the 2i2c Pangeo Hub 🎉

Getting this deployment done was quite an effort that exposed a lot of complexity that wasn't obvious ahead of time. It was a good example of how much care needs to go into crafting cloud infrastructure and operating it over time.

We should write a blog post that briefly describes the major issues that we ran into here, how we got over them, and what kinds of lessons others could learn from the experience.

Value / benefit

• Other hub engineers would benefit by learning from our experience, so that they might be able to more easily anticipate and get over challenges associated with deploying a Pangeo hub.
• People interested in 2i2c would potentially see this blog post as a demonstration of our expertise, as well as the complexity that 2i2c can handle for others

Some major things we've done

It'd be helpful if we kept track of where major discussion happened around this deployment, so that we can use this to generate ideas for topics to cover. A few that I know of:

• Auto-provisioning discussion: Scoping machine sizes for node pools or use auto-provisioning infrastructure#666 (comment)
• Challenges around NFS and Google Filestore expense: Use Google Filestore for home directories infrastructure#651
• Requester pays data: Pangeo Hub access to requester pays data infrastructure#655
• Private nodes: Enabling private nodes for GKE clusters infrastructure#538
• GitHub Teams auth: Authenticate users with GitHub Teams membership in Pangeo Hub infrastructure#598
• Permissive singleuser network policies: Configure a single general purpose and more permissive network policy infrastructure#741

Ideas for blog posts

It'd better if we write a few blog posts about specific things rather than one mega post, so let's brainstorm ideas for topics as they come up:

• The value in using private nodes, and some challenges we had to overcome to make this work (maybe also a meta-point that university cloud projects will run into these challenges in general)
• Deciding whether to use GFS vs. our own hand-rolled filesystems
• Using the Pangeo GitHub Teams auth setup as inspiration for upstreaming this work in JupyterHub
• Creating permissive singleuser network policies that still protect our infrastructure

Tasks to complete

• Continue keeping track of updates as we make them
• Once we are happy with the initial deployment, write up an update post!

Updates

No response

[sgibson91]

Might I suggest this be done in at least 2 parts? We've learned a lot and trying to compress it all into a single blog is probably a reach!

[choldgraf]

Makes sense to me - i just wanted this issue to track the stuff we might talk about 🙂