Better handling of schema change

[utsengar]

If the model is changed, these functions needs modification:

In core.py:

1. def from_backend(self, dagobah_id) (Else it will overwrite the data)
2. Task._serialize

Try to find a better way of managing this.

Also, after a schema change, I am currently deleteing the existing dagobah.db and recreating it. Is there a better way of doing this? schema migrations?

[thieman]

Filing this as a bug. The SQL handling is all pretty crappy and could be made better. One thing I will mention here is that any fixes must work for the Mongo, SQL, and Base backends, so the canonical definitions of the models will have to be stored in either the core classes or in a new centralized module. I think the best way would be for from_backend to do its best to figure out what it wants from the core classes themselves.

To your last question, yes, data should be able to be preserved through schema migrations. I'm guessing SQLAlchemy has some functionality that should help us automate this.

[thieman]

This is blocking the next release. Everyone that uses SQLite is going to have a shitty time upgrading unless the migration handles itself.

[thieman]

Closed by f917990

[thieman]

Whoops, I fixed one issue but not the other. Reopening this until changing the model is less of a pain.

[thieman]

@utkarsh2012 Would appreciate your input here, as I'm having a hard time seeing an easy solution to the model definition problems in core.py. Here's my understanding of what changes in core.py when the Task schema is updated. Let's add a new variable called yoda.

1. Go to Task.__init__ and set yoda to its starting value. If it needs its own setters or special defaults (e.g. soft_timeout), then handle that now, too.
2. If yoda needs to be persisted in the backend, add it to Task._serialize. Importantly, this does not necessarily need to be done for everything in the Task definition, though it will be done for every variable included in the backend models.
3. If yoda can be set on a new Task at creation time, add it to Dagobah._add_job_from_spec (this used to be in from_backend when this issue was originally posted). Again, importantly, this does not necessarily need to be done for every new variable. It also does not necessarily need to be done if you did the previous step.

Given that these three steps aren't necessarily related, I think the only way to consolidate the process would be to add what would probably be a fairly complex config object that is capable of assigning variables to any subset of these three steps, as well as applying formatters to each individual step (e.g. soft_timeout gets a default value of 0 in step 2).

Do you agree? If not, how do you think we could do this? If you do agree, do you think this path is worth it?

[utsengar]

I think addition of Alembic fixes a major issue of schema migration. Existing users will see a flawless migration.

Regarding the changes needed (step 1 to 3) in the app after addition of a variable to a table, as you mentioned it looks like a hard problem to solve which is not really worth tackling. New variables are not really added frequently to a table, when they are added we can go through this exercise.

Although it should be clearly documented some where (in developer doc rather than user doc) so that someone can hack around it quickly.

[thieman]

Added a bit of developer documentation in the wiki: https://github.com/tthieman/dagobah/wiki/Changing-the-Core-Models