Schema and data migrations

[GabeMedrash]

Is your feature request related to a problem? Please describe.

I work in an R&D environment, and even with tons of forethought and discussion regarding what the shape of our data will be, there will always be a day--and always sooner rather than later--that we'll want to change the name of a field, change the type of a field, add a field, etc etc.

The only way I know how to both change the collections configuration in the CMS' config.yml and have already-existing, CMS-produced data reflect that change is to first edit the config.yml, then manually update the markdown/yaml data that has already been produced to match the new schema.

Describe the solution you'd like

I'm not entirely certain what a great solution looks like. One solution that I've been trying to think through is a schema and data migration tool. For example, a tool like flyway for relational databases allow you to define your schema and any modifications to it through ordered migration files. Similarly, a netlify-cms-migration tool could read in a series of migration files in order to produce a single config.yml. Along the way, it would run updates on already-existing data to ensure the data matches the new schema.

I don't believe, however, that a tool that accomplishes the above would be trivial to implement. Are there other ways that folks have accomplished this that works for them?

Describe alternatives you've considered

Continue to do everything by hand?

[erquhart]

I haven't spoken to or heard from anyone who has gone through this kind of migration on content created by Netlify CMS. Knee jerk from 10,000 feet:

• We should be able to do this from the browser using current backend API's, creating one or more commits for the migration.

• Large sites (5k+ entries) will run into rate limits if they're using cloud services like GitHub.com. We could intelligently split migrations into operations that fit within rate limits, tracking the migration metadata together with the accumulated migration content on a branch in the remote.

You're right, this is not trivial. But it's also not crazy. Just takes a lot of consideration.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[erquhart]

It's unorthodox, but I'd like to see this handled through a Config UI, which is on our roadmap - changing a field name through the UI would result in automatic migration behind the scenes. Doing so in a commit makes rolling back migrations trivial.

Related to #341

[eethann]

Noting here that our team recently ran into this on a project and really needed such a migration feature.