Automate production migration on deployment

[afred]

(Copied from #76)

k8s migrations

After some research, a fully automated k8s migration process seems more complicated than we initially anticipated:

Strategy

On deploy:

• (optional) Backup database
• Scale down Chowda to 0 pods
• Perform migrations

On success:

• Update Chowda deployment image
• Scale back up to n pods

On failure:

• Rollback db
• Scale back up to n pods

Problems

This simplistic strategy has some drawbacks:

• Downtime for Chowda (not a big problem)
• Can't be run as init container in Chowda deployment, because there could be multiple instances that try to run

Options

• bash script
  • Using kubectl
  • Run migrations manually, when needed
• Metaflow
  • Running in argo, attached to argo-events
  • Needs special permissions to change kube deployments
• argo-cd
  • Attach to resource_hooks: Sync or PreSync