Data migrations v2 #1956

noirbizarre · 2018-11-16T15:25:25Z

This PR introduce a new migration engine and drops legacy JS migrations.
Migrations are now python based allowing to reuse some udata helpers.

Migration format

A migration file is structured like this:

'''
My migration docstring
'''

def migrate(db):
    db.test.insert_one({'key': 'value'})

Logging

If you want to output some logging, just use the standard python logging facilities

'''
My migration docstring
'''
import logging

log = logging.getLogger(__name__)

def migrate(db):
    log.info('Log some output')

Rollback/revert handling

A migration can optionnaly be reverted. You need to add the optionnal rollback() method:

'''
My migration docstring
'''
import logging

log = logging.getLogger(__name__)

def migrate(db):
    db.test.insert_one({'key': 'value'})
    log.info('Log some output')

def rollback(db):
    db.test.delete_one({'key': 'value'})

Commands

`udata db migrate`

`udata db status`

`udata db info <migration>`

`udata db unrecord <migration>`

Miscellaneous

As a positive side-effect, migrations are not depending anymore on raw Mongo db.eval which:

makes it more secure
makes it compatible with Mongo 4+
allows to run on sandboxed Mongo instances (like Heroku)

Fixes #981

abulte · 2018-11-16T17:25:40Z

We can use udata models in there, right?

abulte

A few typos.

abulte · 2019-01-09T15:40:19Z

CHANGELOG.md

+
+### Breaking changes
+
+- The new migration system ([#1956](https://github.com/opendatateam/udata/pull/1956)) use a new python based format. Pre-2.0 migrations are not compatible so you might need to upgrade to the latest `udata` version `<2.0.0`, execute migrations and then upgrade to `udata` 2+.


s/use/uses/

abulte · 2019-01-09T15:45:16Z

udata/commands/db.py

+@click.argument('filename', default=None, required=False, metavar='[FILENAME]')
+def info(plugin_or_specs, filename):
+    '''
+    Display detailled info about a migration


s/detailled/detailed/

abulte · 2019-01-09T15:48:12Z

udata/migrations/__init__.py

+        '''
+        Status is the status of the last operation.
+
+        Will be `None` is the record doesn't exists.


s/exists/exist/

abulte · 2019-01-09T15:50:24Z

udata/migrations/__init__.py

+                        self.module.rollback(db)
+                        out = _extract_output(q)
+                        self.add_record('rollback', out, db._state, True)
+                        msg = 'Error while executing migration, rollback has been applyied'


s/applyied/applied/

abulte · 2019-01-09T15:51:06Z

udata/migrations/__init__.py

+        return out
+
+    def unrecord(self):
+        '''Only record a migration without applying it'''


Wrong docstring?

abulte · 2019-01-09T15:51:40Z

udata/migrations/__init__.py

+    return Migration(plugin, filename)
+
+
+def list_availables():


s/list_availables/list_available/

abulte · 2019-01-09T15:51:58Z

udata/migrations/__init__.py

+    '''
+    List available migrations for udata and enabled plugins
+
+    Each row is tuple with following signature:


s/tuple/a tuple/

abulte · 2019-01-09T15:52:15Z

udata/migrations/__init__.py

+
+def _iter(plugin, module):
+    '''
+    Iter over migrations for a given plugin module


s/Iter/Iterate/

abulte · 2019-01-09T15:52:31Z

udata/migrations/__init__.py

+
+
+def _module_name(plugin):
+    '''Get the module namefor a given plugin'''


s/namefor/name for/

### Breaking changes - Migration to Python 3.7 [#1766](#1766) - The new migration system ([#1956](#1956)) uses a new python based format. Pre-2.0 migrations are not compatible so you might need to upgrade to the latest `udata` version `<2.0.0`, execute migrations and then upgrade to `udata` 2+. - The targeted mongo version is now Mongo 3.6. Backward support is not guaranteed ### New features - New migration system [#1956](#1956): - Use python based migrations instead of relying on mongo internal and deprecated `js_exec` - Handle rollback (optionnal) - Detailled history - Template hooks generalization: allows to dynamically extend template with widgets and snippets from extensions. See [the dedicated documentation section](https://udata.readthedocs.io/en/stable/extending/#hooks) [#2323](#2323) - Markdown now supports [Github Flavored Markdown (GFM) specs](https://github.github.com/gfm/) (ie. the already supported [CommonMark specs](https://spec.commonmark.org) plus tables, strikethrough, autolinks support and predefined disallowed raw HTML) [#2341](#2341)

noirbizarre added 5 commits November 15, 2018 14:32

Drop legacy JS migrations

17fc7c9

WIP

401020b

WIP

776bc20

WIP

aeade57

WIP

a8ebb03

noirbizarre added enhancement documentation refactoring to-be-discussed labels Nov 16, 2018

noirbizarre added this to the 2.0 milestone Nov 16, 2018

noirbizarre added 9 commits November 28, 2018 10:19

WIP

659f9f4

Flush cache

03bc40a

Upgrade pip during build

32c43c4

Introduce RollbackError

917b35e

Improve Record

dc3188b

Refactor

b5933a6

More cosmetics

674f187

Fixes and refactor

ad037b8

Added changelog

cd96b96

noirbizarre changed the title ~~[WIP] Data migrations v2~~ Data migrations v2 Jan 7, 2019

Pin faulty har-validator dependency

e442f14

abulte approved these changes Jan 9, 2019

View reviewed changes

Post review fixes

cd8e806

noirbizarre merged commit 7e15c97 into opendatateam:py3 Jan 9, 2019

noirbizarre deleted the data-migrations-v2 branch January 9, 2019 21:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Data migrations v2 #1956

Data migrations v2 #1956

noirbizarre commented Nov 16, 2018 •

edited

abulte commented Nov 16, 2018

abulte left a comment

abulte Jan 9, 2019

abulte Jan 9, 2019

abulte Jan 9, 2019

abulte Jan 9, 2019

abulte Jan 9, 2019

abulte Jan 9, 2019

abulte Jan 9, 2019

abulte Jan 9, 2019

abulte Jan 9, 2019

abulte Jan 9, 2019


		### Breaking changes

		- The new migration system ([#1956](https://github.com/opendatateam/udata/pull/1956)) use a new python based format. Pre-2.0 migrations are not compatible so you might need to upgrade to the latest `udata` version `<2.0.0`, execute migrations and then upgrade to `udata` 2+.



		def _module_name(plugin):
		'''Get the module namefor a given plugin'''

Data migrations v2 #1956

Data migrations v2 #1956

Conversation

noirbizarre commented Nov 16, 2018 • edited

Migration format

Logging

Rollback/revert handling

Commands

udata db migrate

udata db status

udata db info <migration>

udata db unrecord <migration>

Miscellaneous

abulte commented Nov 16, 2018

abulte left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noirbizarre commented Nov 16, 2018 •

edited

`udata db migrate`

`udata db status`

`udata db info <migration>`

`udata db unrecord <migration>`