Dev cluster from cloudops way

[imbstack]

HEADS UP: This changes the name of the azure account env var (among other things) in most services. updated heroku before landing!

This takes the new way of doing things with helm and brings into the tc repo development flow a bit more.

1. yarn generate builds a bunch of helm templates
2. it also makes a schema for config options, an example config file, and defaults
3. adds some niceties around helm deployments and such
4. Moves entrypoint generation out of the build process and checks it in
5. some stuff with the dockerfile I think I'll undo
6. add the generated stuff to the gitattributes list of things to hide

Hey look, it Works ™️ https://taskcluster.imbstack.com/

I still need to fill out 90% of the variables for services other than auth, but at least auth responds to ping and the ui shows up.

Heads up to @djmitche @owlishDeveloper @edunham @sciurus. Thanks to the cloudops folks, we could never have gotten this correctly without your guidance.

Things TODO:

• Make the helm stuff in yarn scripts a bit more correct. It fails to update properly sometimes at the moment (ANSWER: Helm3 currently has a bug where upgrades don't work. For the shared dev cluster we can either wait for helm3 to fix itself or securely install helm2)
• Figure out what we're going to do with terraform bits. We could replace them all with a script and forgo terraform entirely? (ANSWER: Get rid of them after cloudops takes them over)
• Set up some sort of yarn dev:init that creates a config file for you with many of the per-service fields filled out such as rabbitmq passwords and such. The actual work of creating things would either be done by the script or perhaps terraform.
• Set up a single medium-sized dev cluster for the taskcluster team to use. Each developer would have their own namespace there in which to deploy their taskcluster. This should both save us money and operational effort.
• Figure out if this will actually work with the cloudops deployment stuff as-is or if there are things we'll need to tweak. (ANSWER: It seems like this will work with what they have. It will be a bit of work on their side but it can work)
• Optionally get this working with minikube. I think the only thing we'd need is to make two ingress files and only apply one in helm depending on some sort of variable
• various cleanups and papercuts. This was made in a bit of a frenzy so lots of things could be renamed or cleared up.
• Add docs on how to use this!

This is a huge diff but a lot of it is autogenerated so you can mostly ignore it. Things of interest:

• infrastructure/k8s/values.schema.json: helm 3 supports a schema file for input values natively! Right now the schema I've generated is a bit easy-going because our underlying configs don't do a good job of indicating that a config option is optional. I'd like to start where we are now and then tighten over time.
• user-config-example.yaml: This probably needs a new name, but it is a list of every option you can pass into a taskcluster instance. At some point this should grow descriptions.

[djmitche]

Looking good!

[sciurus]

Right now AZURE_ACCOUNT, FORCE_SSL, TRUST_PROXY, PULSE_HOSTNAME, PULSE_NAMESPACE, PULSE_VHOST, TASKCLUSTER_ROOT_URL are all specified explicitly for each service.

I think these are all global values that could live at the top-level like dockerImage does. Would that be feasible to implement? As it currently stands you have to duplicate those values a lot.

(Tangentially, related to AZURE_ACCOUNT, https://bugzilla.mozilla.org/show_bug.cgi?id=1566610)

[imbstack]

Ok, this is now "entirely working".

Some services don't run due to small config details that we still need to set up. I have faith that we can track these down in short order -- I am just tired of doing the tracking so hopefully someone can help me there.

There's also some quite manual hacks in the dev cluster deployment stuff. Hopefully someone who understands these services better than I can will help un-hack.

Other than that (and the fact that helm3 doesn't do updates correctly yet) I think this is actually usable!

[djmitche]

A bunch of pretty minor comments here, but this is looking good and I'm excited to use it.

[sciurus]

Thanks for adopting my suggestions. I took another look and still don't see any obvious problems with us using this. I did a brief test rendering the k8s templates for github using the helm templates from this PR and it worked fine.

🚢

[imbstack]

Ok, I have an entirely working dev deployment from this now!

I'm going to update the heroku configs and land this first thing tomorrow.