Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ponyorm migration tool #16

Open
steinster opened this issue Aug 7, 2013 · 56 comments
Open

ponyorm migration tool #16

steinster opened this issue Aug 7, 2013 · 56 comments
Assignees

Comments

@steinster
Copy link

@steinster steinster commented Aug 7, 2013

Hello,

I'm not sure if this is the place to put this question. If there is a better place please let me know.

Is there a DB migration tool for pony orm? Something like alembic for sqlalchemy..

@amalashkevich
Copy link
Member

@amalashkevich amalashkevich commented Aug 7, 2013

Hi,
There is no such a tool yet, although we are working on it. Will let you know when it is ready. Thank you for asking.

On Aug 7, 2013, at 11:51 AM, steinster notifications@github.com wrote:

Hello,

I'm not sure if this is the place to put this question. If there is a better place please let me know.

Is there a DB migration tool for pony orm? Something like alembic for sqlalchemy..


Reply to this email directly or view it on GitHub.

@steinster
Copy link
Author

@steinster steinster commented Aug 7, 2013

If you need any help with it I would love to help with the project a bit.

@amalashkevich
Copy link
Member

@amalashkevich amalashkevich commented Aug 7, 2013

Thank you, I appreciate it, we'll get in touch.

On Aug 7, 2013, at 7:27 PM, steinster notifications@github.com wrote:

If you need any help with it I would love to help with the project a bit.


Reply to this email directly or view it on GitHub.

@ghost ghost assigned amalashkevich Aug 30, 2013
@amalashkevich
Copy link
Member

@amalashkevich amalashkevich commented Aug 30, 2013

Hi,
We have created the mailing list ponyorm-list@ponyorm.com. This is the place to ask questions, share experiences, or discuss the Pony ORM with other users and the Pony ORM developers. You can subscribe to the list at http://ponyorm-list.ponyorm.com
Regards,
Alexey

@guyskk
Copy link

@guyskk guyskk commented Oct 30, 2015

+1

1 similar comment
@NiklasRosenstein
Copy link

@NiklasRosenstein NiklasRosenstein commented Nov 27, 2015

+1

@andreymal
Copy link

@andreymal andreymal commented Mar 21, 2016

Even third-party tools are still not exist?

@nadermx
Copy link

@nadermx nadermx commented May 12, 2016

Any specific parts I can try and help with getting this done? I need migrations pretty badly at this point

@kozlovsky
Copy link
Member

@kozlovsky kozlovsky commented May 13, 2016

Right now we are working on release with full JSON support, and after that we will switch to working on migrations. Hope the initial version will be released soon after that..

@Vic020
Copy link

@Vic020 Vic020 commented Jun 20, 2016

+1
get in touch

@hackrole
Copy link

@hackrole hackrole commented Jun 21, 2016

+1, waiting for your staff

@misakar
Copy link

@misakar misakar commented Aug 7, 2016

+1

@nadermx
Copy link

@nadermx nadermx commented Sep 14, 2016

@kozlovsky I saw that you pushed the json update! Any word or any suggestions on how I could contribute with some code in the method you have in mind in order to get migrations done quicker. Having to do manual alter tables and indexing is getting to be cumbersome

@darkf
Copy link

@darkf darkf commented Oct 14, 2016

Yeah, this is the main missing feature of Pony right now. It's excellent otherwise, but really needs migrations (preferably automatic).

@pingf
Copy link

@pingf pingf commented Oct 19, 2016

really needed!

@sighalt
Copy link

@sighalt sighalt commented Nov 28, 2016

+1

2 similar comments
@TimothyBramlett
Copy link

@TimothyBramlett TimothyBramlett commented Nov 29, 2016

+1

@pylover
Copy link

@pylover pylover commented Nov 29, 2016

+1

@danielhrisca
Copy link

@danielhrisca danielhrisca commented Nov 30, 2016

Any progress with this topic?

@d-me-tree
Copy link

@d-me-tree d-me-tree commented Dec 3, 2016

+1

@darkf
Copy link

@darkf darkf commented Dec 3, 2016

@amalashkevich @kozlovsky Any status update on this? There are approximately a kajillion people waiting on and +1ing this, it would be nice to see where it's going or if we should migrate (pun intended) to another ORM with this functionality.

@kozlovsky
Copy link
Member

@kozlovsky kozlovsky commented Dec 3, 2016

The core migration functionality is done, we need approximately two weeks to release the first version

@darkf
Copy link

@darkf darkf commented Dec 4, 2016

Thanks for the update! Good to hear.

@danielhrisca
Copy link

@danielhrisca danielhrisca commented Dec 4, 2016

Great news! Thank you for your work!

@famargar
Copy link

@famargar famargar commented Dec 12, 2016

Keep us posted!

@kafeg
Copy link

@kafeg kafeg commented Dec 13, 2016

Hi! Very important feature! Waiting for release)

@pingf
Copy link

@pingf pingf commented Feb 22, 2017

'approximately two weeks' becomes two months 😞

@darkf
Copy link

@darkf darkf commented Feb 22, 2017

Yeah, any word on this feature @kozlovsky?

@kozlovsky
Copy link
Member

@kozlovsky kozlovsky commented Feb 22, 2017

Hi guys, the task of writing good migration tools turns out to be more complex than I expected.

The delay was caused by the following reason: in current version of Pony some schema names are incompatible with a future migration tool.

One example is two entities Foo and Bar which have a many-to-many relationship. That is, Foo class has a bars attribute of type Set, while Bar class has a foos attribute of the same type. In that case, current version of Pony names underlying many-to-many table as bar_foo (that is, it used the names of two entities sorted in alphabetic order). This works if there are only one many-to-many attribute between this pair of entities. If there are several such attributes, they become called bar_foo_2, bar_foo_3, etc. This is very error-prone when using migrations, because introducing new many-to-many attribute between the same two entities can change the meaning of all bar_foo_N intermediary tables, which lead to corrupted data.

Another problematic situation is with a long names which don't fit within specified database maximum object name length. Current version of Pony just cut such names to maximum allowable length, but this can lead to situation when several different columns/foreign keys/indexes has the same name. The most problematic database in this aspect is Oracle, because in it the maximum name length cannot be greater than 30 chars. So even several independent columns can be erroneously cut to the same name. PostgreSQL and MySQL has less restrictive constraints of 63 and 64 max symbols in name, but even that may be not enough for automatically generated names of foreign keys and composite indexes. The correct solution is to add some unique md5 hash to the end of a truncated identifier name, if the length of the original name is too big.

The migration-enabled version of Pony will handle the names of tables, columns, foreign keys and indexes in that enhanced way. But what to do with existing databases? We think they should be upgraded to this new naming standard before migrations can be applied to them. In order to do that, we add a new system table pony_version, and if that table is not exists, we interpret that as that the database was used with pre-migrations version of Pony, and in that case new Pony refuses to work with that database, until an upgrade was performed. The implementation of that upgrade was rather complex, because many different types of objects should be renamed: if some table is renamed, we need to rename its foreign keys and indexes as well, and in different databases it requires different SQL commands.

At this moment we have correct implementation of that upgrade logic for all supported databases, and also implementation of the migration tool. What rest is to combine upgrade logic and migration logic together and propose some API which will be convenient to user. We plan to replace db.bind(**settings.db_params) and db.generate_mapping(create_tables=True) database methods to db.connect(**settings.db_params) method and another db.migrate(**settings.db_params) method which can handle command-line arguments such as migrate make, migrate apply and migrate showsql, but this part of work requires several more days to be implemented correctly.

Resuming, we actively working on migration tool, but my estimates about the time it requires to be implemented turn out to be completely incorrect. I think the work on migration tool is finished on 90% at this moment, but it will be unrealistically to say we can release migrations during this week. I really hope we can release it during the following week, but as history has shown, it is very hard to give correct estimations about the time required for implementing of such a complex tool as migration support. But we are working on it, and I think the final is already close.

@pingf
Copy link

@pingf pingf commented Apr 7, 2017

any news on this feature?

@luishdzupiita
Copy link

@luishdzupiita luishdzupiita commented Jul 27, 2017

any news on this feature? +1

@danielhrisca
Copy link

@danielhrisca danielhrisca commented Jul 27, 2017

Many are waiting nervously for this feature to come to pony. I hope it's not going to take much longer

@kickapoo
Copy link

@kickapoo kickapoo commented Oct 15, 2017

In my case this is the only thing that block me for full pass to pony.
Please @kozlovsky if you could give another feedback about the current state will be most appriciated.

@kozlovsky
Copy link
Member

@kozlovsky kozlovsky commented Oct 16, 2017

At this moment we have some internal branch which can handle simple migrations (like, adding/deleting entities, attributes and relationships). I hope we can push it to GitHub really soon. But before that we need to do some final review during this week.

@johanngomes
Copy link

@johanngomes johanngomes commented Nov 7, 2017

Hi, @kozlovsky and team. Any news on when we can have this migration functionality? Thanks,

@kozlovsky
Copy link
Member

@kozlovsky kozlovsky commented Nov 8, 2017

Hi, I just pushed a new branch orm-migrations with the first version of PonyORM migrations. You can read the description of API here: https://github.com/ponyorm/pony/tree/orm-migrations/pony/migrate

The code is very raw at this point and not intended for production usage yet, we still need to work on it. I think really simple use-cases for PostgreSQL and MySQL should work, but SQLite support is limited, and merge functionality is untested. Adding entity or attribute should work, but, say, changing inheritance hierarchy or modifying a type of relationship will raise NotImplementedError. Anyway, I hope the working version of migrations is close.

@johanngomes
Copy link

@johanngomes johanngomes commented Nov 8, 2017

Thank you for your update, @kozlovsky !

@hammeronthenet
Copy link

@hammeronthenet hammeronthenet commented Dec 1, 2017

Hi @kozlovsky,
nice!

One question: what's better to install 0.8 version? When do you think 0.8 will be available on pip?

Thank you!

@zxytim
Copy link

@zxytim zxytim commented Jun 25, 2018

How's migration going now?

@patrickporto
Copy link

@patrickporto patrickporto commented Jun 26, 2018

In moment has some alternative for schema migrations on PonyORM?

@nadermx
Copy link

@nadermx nadermx commented Jun 26, 2018

I can confirm that migrations is working, if you want to install it via pip currently do the following

pip install git+git://github.com/ponyorm/pony.git@orm-migrations

All the documentation on how to do migration can be found here

Also if you are using pony with flask blueprints, please note ref another issue

generate_mapping call from app/init.py to some submodule like app/main.py which is not imported when from app import models line of migrate.py module is executed

@japrogramer
Copy link

@japrogramer japrogramer commented Sep 1, 2018

what is the status of this orm-migrations? when will it be merged upstream.

@dibstern
Copy link

@dibstern dibstern commented Oct 3, 2018

+1
Pony seems awesome, but I can’t use an ORM without a reliable migrations tool. Super keen for this to be released and to start using Pony

@Elijas
Copy link

@Elijas Elijas commented Jan 7, 2019

The way I currently solve the migration problem is by partly using a SQL migration script generator (Liquibase at the moment).

I create two schemas with ponyorm, then run them apply them on two different databases, then run the SQL diff tool to generate SQL code (ALTER, etc.)

Any additional migration of data itself is done within the python, around the SQL code

@Elijas
Copy link

@Elijas Elijas commented Jan 7, 2019

By the way - is a subset of migration scripts already ready for production code? Even that would be great

@japrogramer
Copy link

@japrogramer japrogramer commented Jan 7, 2019

@Elijas sounds intense. lots of steps to introduce an error.
Would prefer a similar migration layout to the django orm.

@Elijas
Copy link

@Elijas Elijas commented Jan 8, 2019

@Elijas sounds intense. lots of steps to introduce an error.
Would prefer a similar migration layout to the django orm.

I think so too, but I kinda really want to keep ponyorm :D

To make it less error-prone, I'm currently considering using

Are there any other good alternatives to yoyo? For the schema-migrations, as I'd rather migrate data with pony

@Elijas
Copy link

@Elijas Elijas commented Jan 8, 2019

I've searched some info about migrations, and why it's a difficult task - I think this article summarizes it best, though you guys might want to look at it, to acquire some sympathy for the @ steinster

@zxytim
Copy link

@zxytim zxytim commented May 3, 2019

How's migration going now? It's been more than a year since my last check.
Is there any roadmap that migration will be merged into orm branch, or the release of 0.8.0?
Really looking forward to the official release of migration function.
@kozlovsky Is there anything we could do help accelerate this process? How can we help you?

@Elijas
Copy link

@Elijas Elijas commented May 3, 2019

@sostholmwork
Copy link

@sostholmwork sostholmwork commented Apr 28, 2020

Any updates on this? :)

@fluffy-critter
Copy link

@fluffy-critter fluffy-critter commented Jun 29, 2020

Even without a full migration tool, is there a mechanism that can be used to do simple column and foreign key additions to a schema in a fairly DB-independent manner? Like the problem seems to become much simpler if you don’t have to worry about renaming or changing data types or the like.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.