Support schema migration

[artgoldberg]

Stored Models, such as spreadsheets and delimited files on disk, can become incompatible with updates to model definitions in obj_model. Create a utility that migrates stored models to be compatible with a modified model. E.g., Django has a migration utility, which was originally called fabric.

[artgoldberg]

We would currently benefit from a simple tool that migrates an obj_model file from one data Model definition to another. We've discussed this before and not made such a tool, but I think the need for it has grown.

First, the number of active wc_lang model files has grown substantially. E.g., wc_sim now has 46, most of which are validation cases. I expect this trend will continue. Second, we're in the process of changing some data Models in wc_lang: removing k_cat & k_m from rate laws, adding type to compartments, etc.

Let's start with a simple migration tool that handles one data Model at a time. The obviously useful migrations are 'add attribute' (perhaps with default value), and 'delete attribute'. There are many other potentially useful migrations of a single data Model, such as 'write attributes in order', 'change attributes order', 'change attribute default value', and 'change attribute name', and many possible migrations involving two data Models, but they're not needed now, as far as I know.

[jonrkarr]

I agree with the need. For now, I've compiled a list of the wc-lang-encoded files and am migrating them manually. I've also added some tests, particularly in wc-sim to check that the files are up-to-date with respect to wc-lang. If I can figure out a way to have two versions of wc-lang instantiated simultaneously, this would be pretty easy to implement, at least for the basic cases you outlined. Jonathan

…

On Sat, Nov 17, 2018 at 9:16 AM artgoldberg ***@***.***> wrote: We would currently benefit from a simple tool that migrates an obj_model file from one data Model definition to another. We've discussed this before and not made such a tool, but I think the need for it has grown. First, the number of active wc_lang model files has grown substantially. E.g., wc_sim now has 46, most of which are validation cases. I expect this trend will continue. Second, we're in the process of changing some data Models in wc_lang: removing k_cat & k_m from rate laws, adding type to compartments, etc. Let's start with a simple migration tool that handles one data Model at a time. The obviously useful migrations are 'add attribute' (perhaps with default value), and 'delete attribute'. There are many other potentially useful migrations of a single data Model, such as 'write attributes in order', 'change attributes order', 'change attribute default value', and 'change attribute name', and many possible migrations involving two data Models, but they're not needed now, as far as I know. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACt2KavwKc5cPTzO_JuOWTeoXM8OyLvUks5uwBpRgaJpZM4RuTov> .

[artgoldberg]

Since I've the biggest need I'll take a crack at it today. (You don't see many of wc_sim's wc-lang-encoded files because I've not pushed them yet. I'm working towards that with due haste.)

I think it will be straightforward to instantiate two versions of wc-lang simultaneously by putting them in separate directories and using the directory hierarchy to create distinct names for corresponding Models. The instructions would look like this:

To migrate a model file from one wc_lang version to another (one set of obj_models to another) create separate repos with the two wc_lang versions, like this:

    wc_lang_old/wc_lang     # an old wc_lang repo
    wc_lang_new/wc_lang     # a new wc_lang repo

and then execute

    migrate wc_lang_old wc_lang_new model_file_to_migrate.xlsx ...

This will read model_file_to_migrate.xlsx using the models in wc_lang_old, determine the differences between the models in wc_lang_old and wc_lang_new, change the model obtained from model_file_to_migrate.xlsx into the structure of wc_lang_new, and then write it into model_file_to_migrate-wc_lang_new.xlsx. Corresponding models in wc_lang_old and wc_lang_new must have the same class names.

Only certain types of differences between models in wc_lang_old and wc_lang_new are supported:
Add or remove an attribute
Change the order of attributes in a model
Migration that involves coordinated changes among multiple models are not supported.

[jonrkarr]

The functionality will likely resemble the copy method.

…

On Sun, Nov 18, 2018, 7:51 AM artgoldberg ***@***.*** wrote: Since I've the biggest need I'll take a crack at it today. (You don't see many of wc_sim's wc-lang-encoded files because I've not pushed them yet. I'm working towards that with due haste.) I think it will be straightforward to instantiate two versions of wc-lang simultaneously by putting them in separate directories and using the directory hierarchy to create distinct names for corresponding Models. The instructions would look like this: To migrate a model file from one wc_lang version to another (one set of obj_models to another) create separate repos with the two wc_lang versions, like this: wc_lang_old/wc_lang # an old wc_lang repo wc_lang_new/wc_lang # a new wc_lang repo and then execute migrate wc_lang_old wc_lang_new model_file_to_migrate.xlsx ... This will read model_file_to_migrate.xlsx using the models in wc_lang_old, determine the differences between the models in wc_lang_old and wc_lang_new, change the model obtained from model_file_to_migrate.xlsx into the structure of wc_lang_new, and then write it into model_file_to_migrate-wc_lang_new.xlsx. Corresponding models in wc_lang_old and wc_lang_new must have the same class names. Only certain types of differences between models in wc_lang_old and wc_lang_new are supported: Add or remove an attribute Change the order of attributes in a model Migration that involves coordinated changes among multiple models are not supported. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#6 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACt2KULpmBF4bmtfa1IXBdpb6-QzSz9Pks5uwVe_gaJpZM4RuTov> .

[artgoldberg]

Yes, it's similar to obj_model.core.Model.copy(), but that only copies one Model instance, not a network of them. Migrating a network will look somewhat like the pseudocode below. to make it easy to connect up the related attributes in the migrated network, in each old_model I put a reference (_migrated_copy) to its corresponding migrated copy. I'm completely ignoring more complex migrations, like breaking a Model into two, etc.

Is there code in obj_model i can reuse for this? I can't find any.

def deep_migrate(old_models, old_schema, new_schema):
    '''
    supports:
        delete attributes in old_schema (but not primary keys)
        add attributes in new_schema
        add model definitions in new_schema
        assumes that otherwise the schemas are identical
    old_models: list
    old_schema: dict: model_name -> model_def
    new_schema: dict: model_name -> model_def
    '''
    all_models = []
    for old_model in old_models:
        cls_name = old_model.__class__.__name__
        new_model = migrate_model(old_model, old_schema[cls_name], new_schema[cls_name])
        all_models.append(old_model, new_model)
    connect_models(all_models)

def migrate_model(old_model, old_model_def, new_model_def):
    '''
    old_model: obj_model.Model 
    old_model_def: def of the old model
    new_model_def: def of the old model
    '''
    new_model = new_model_def()
    copy data attrs that weren't deleted from old_model to new_model
    for new_attrs in new_model_def:
        assign default if defined
    old_model._migrated_copy = new_model
    return new_model

def connect_models(all_models):
    # all_models: list of pairs of corresponding old and new models
    for old_model, new_model in all_models:
        for related_attr in new_model:
            update related_attr to old_model.related_instance._migrated_copy

[jonrkarr]

There's no code that does exactly this. I think `copy` is the most similar method. You do want to migrate a model and all of its related objects.

…

On Tue, Nov 20, 2018 at 10:14 AM artgoldberg ***@***.***> wrote: Yes, it's similar to obj_model.core.Model.copy(), but that only copies one Model instance, not a network of them. Migrating a network will look somewhat like the pseudocode below. to make it easy to connect up the related attributes in the migrated network, in each old_model I put a reference (_migrated_copy) to its corresponding migrated copy. I'm completely ignoring more complex migrations, like breaking a Model into two, etc. Is there code in obj_model i can reuse for this? I can't find any. def deep_migrate(old_models, old_schema, new_schema): ''' supports: delete attributes in old_schema (but not primary keys) add attributes in new_schema add model definitions in new_schema assumes that otherwise the schemas are identical old_models: list old_schema: dict: model_name -> model_def new_schema: dict: model_name -> model_def ''' all_models = [] for old_model in old_models: cls_name = old_model.__class__.__name__ new_model = migrate_model(old_model, old_schema[cls_name], new_schema[cls_name]) all_models.append(old_model, new_model) connect_models(all_models) def migrate_model(old_model, old_model_def, new_model_def): ''' old_model: obj_model.Model old_model_def: def of the old model new_model_def: def of the old model ''' new_model = new_model_def() copy data attrs that weren't deleted from old_model to new_model for new_attrs in new_model_def: assign default if defined old_model._migrated_copy = new_model return new_model def connect_models(all_models): # all_models: list of pairs of corresponding old and new models for old_model, new_model in all_models: for related_attr in new_model: update related_attr to old_model.related_instance._migrated_copy — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#6 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACt2KZLSoquNdme0QtzT_NMaWQ6cSDzDks5uxBxPgaJpZM4RuTov> .

[artgoldberg]

As I expected, and of course.

[jonrkarr]

Initial version of migration is done