Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate production migration on deployment #108

Open
Tracked by #76
afred opened this issue Aug 9, 2023 · 0 comments
Open
Tracked by #76

Automate production migration on deployment #108

afred opened this issue Aug 9, 2023 · 0 comments
Labels
maintenance 🔧 Updates and upgrades old maintenance 🔧 Old label for Updates and upgrades

Comments

@afred
Copy link
Contributor

afred commented Aug 9, 2023

(Copied from #76)

k8s migrations

After some research, a fully automated k8s migration process seems more complicated than we initially anticipated:

Strategy

On deploy:

  • (optional) Backup database
  • Scale down Chowda to 0 pods
  • Perform migrations

On success:

  • Update Chowda deployment image
  • Scale back up to n pods

On failure:

  • Rollback db
  • Scale back up to n pods

Problems

This simplistic strategy has some drawbacks:

  • Downtime for Chowda (not a big problem)
  • Can't be run as init container in Chowda deployment, because there could be multiple instances that try to run

Options

  • bash script
    • Using kubectl
    • Run migrations manually, when needed
  • Metaflow
    • Running in argo, attached to argo-events
    • Needs special permissions to change kube deployments
  • argo-cd
@mrharpo mrharpo added the old maintenance 🔧 Old label for Updates and upgrades label Aug 9, 2023
@singlesoliloquy singlesoliloquy added this to the Planned App Launch milestone Aug 11, 2023
@mrharpo mrharpo added the maintenance 🔧 Updates and upgrades label Aug 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintenance 🔧 Updates and upgrades old maintenance 🔧 Old label for Updates and upgrades
Projects
Status: 📋 Backlog
Development

No branches or pull requests

3 participants