Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dump/Load Metabase DB to/from H2 file #10662

Open
wants to merge 35 commits into
base: master
from

Conversation

@ogeagla
Copy link
Member

commented Aug 21, 2019

We want to be able to dump an existing Metabase database to a H2 file format, with utilities to be able to encrypt, upload, download, and decrypt the H2 file, so that we can easily chain this to exsiting load-from-h2 command to have a complete migration.

The way this will work:

  1. Customer purchases hosted instance
  2. Metabase ops target that hosted instance for migration, run initialize methods to generate s3-upload-url
  3. Provide s3-upload-url to customer so they can run (either through command or API) dump-to-h2 and secure-dump-and-upload
  4. Once customer has successfully run the upload, Metabase ops invokes methods to trigger download and ingest of secure H2 db on the customer's purchased hosted instance.
  • command dump-to-h2 which takes a source (PG/MySql) db and H2 target filename
  • command download-and-unlock to download and unlock a dump at a public S3 location with provided AES secret
  • command secure-and-upload which generates an AES secret and encrypts an H2 dump before it is uploaded to the provided S3 upload url
  • API endpoints for commands: dump-to-h2, secure-and-upload-dump, download-and-unlock-dump, load-from-h2
  • uploads to S3
  • if Metabase is already running on H2, copy the db over to new filename
  • override dupe db entries on ingest
  • compute and save MD5 digest
ogeagla added 2 commits Aug 21, 2019

@ogeagla ogeagla requested a review from camsaul as a code owner Aug 21, 2019

@ogeagla ogeagla changed the title Dump Metbase DB to H2 file Dump Metabase DB to H2 file Aug 21, 2019

ogeagla added 2 commits Aug 21, 2019
@camsaul

This comment has been minimized.

Copy link
Member

commented Aug 21, 2019

@ogeagla let me know when this is ready to be reviewed

I think the most important thing for me is that we have some tests that verify everything works as expected end-to-end with a database with some real data in it. The CI steps that test migrating from H2 -> MySQL and H2 -> Postgres are probably good starting points, you can maybe use that same test DB and then run it in reverse and see if we get the same rows in the original H2 DB and the resulting one

@ogeagla ogeagla force-pushed the og-instance-migration branch from 8860656 to 3b6289a Aug 23, 2019

@camsaul
Copy link
Member

left a comment

One thing we'll definitely need to do before merging is make sure we have a good testing story around this

walterl and others added 10 commits Aug 30, 2019
Upper case column names
H2 (output db) column names are case sensitive and, for some unknown
reason, the H2 db is created with upper cased names, and `cols` are lower-
cased.

@ogeagla ogeagla changed the title Dump Metabase DB to H2 file Dump/Load Metabase DB to/from H2 file Sep 4, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.