Migrate data from v0.38 format to v1.0

[carolynvs]

The entire data storage format and allowed backends is changing for Porter v1. We are moving from flat file storage (without query support) to mongodb. So both the backend plugins are changing AND the storage format.

🚨 Migrations between alpha versions are not supported. Do not use the v1 alpha of Porter with production data.

After #1684 is merged, we cannot release a prerelease of v1 until we have a migration for existing data.

• The migration is one way and does not alter your existing data.
• We need to pick new storage schema versions for our data, and decide where to encode the porter schema version vs the cnab schema version.

One option is to store the porter schema version on the document, and then always generate the cnab version in the new ToCNAB() functions on our document types.

For this situation I think we will need two commands to move old data to the new format:

1. porter storage copy --source myazuretables --destination myazuremongo - Copies the data AS IS, from one storage backend to another. This gets the data out of azure tables and into a mongo database.
2. porter storage migrate - Executes a data migration on the data in the currently configured storage backend. Data in the old format that has been migrated is deleted during the migration, leaving a clean set of documents.

We also require a documentation page and blog post announcing EARLY before the v1 release how the data migrations will work.

[carolynvs]

Work on this can finally start. The last feature that changes the database schema is #1027 (still in progress) so once that is merged, we can implement migrate.

[carolynvs]

We need to do a bit of a trick for this migration since from v0.38.* -> v1.0.0-beta.1+ we have changed the storage plugin protocol. The first prerelease that supports migration will be beta.1.

Here is the process that I'm considering:

1. The user installs porter 1.0.0-beta.1+. Even though the old plugins are not supported with v1, they should keep them installed for use during the migration. The last supported version of the azure plugin is v0.11.2. So don't upgrade the azure plugin yet!
2. They add a storage entry for a new mongodb database which will hold their data going forward after the migration.
3. They change the default storage account to the new mongodb storage entry. Keep the old storage account entries that use deprecated plugins (e.g. azure blob or table storage).

    default-storage = "new-mongo"
     
   [[storage]]
     name = "old-azure"
     plugin = "azure.table"
   
   [[storage]]
     name = "new-mongo"
     plugin = "mongodb"

4. Run porter storage migrate --src old-azure --dest new-mongo.
5. They can now remove the storage entry for old-azure, and upgrade the azure plugin to the most recent version (which drops support for the storage plugin protocol and only supports secrets).

I don't think we will need a porter storage copy command and can just use migrate. From 1.0.0 onward, since we don't make changes to the storage schema without bumping the major version of porter, migrate only ever migrates from N-1 to N (e.g. v0.38 -> 1.0.0-beta.1+, or in the future 1.0.0 -> 2.0.0). So each version of porter knows how to migrate from the previous version to the current version and doesn't need to support multiple migration paths.

The migrate command will use a subset of the old storage protocol to read the data from the previous version out of the old storage account. The original json documents are represented internally as map[string]interface{}. Porter then converts those maps into the new data structures (Installations, Claims, Runs, etc) and persists them to the new storage account.

[github-actions]

Closed by #2150.