Skip to content

downgrade with depedencies leaves revision table in broken state #789

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
hcpatel opened this issue Feb 1, 2021 · 6 comments
Closed

downgrade with depedencies leaves revision table in broken state #789

hcpatel opened this issue Feb 1, 2021 · 6 comments

Comments

@hcpatel
Copy link

hcpatel commented Feb 1, 2021

Describe your question
My use case is 2 branches with their own base to start with, where subsequent revisions from BranchB will depend on BranchA. Each new revision on BranchB I always specify a 'depends on' BranchA (usually --depend_on=branchA@head). But I an finding an issue upgrading to the head of BranchB after I run a downgrade of the latest revision on BranchB (using alembic upgrade heads).
Initially I thought it could be an issue with downgrade, but after some testing realised perhaps it the 'mis-use' of 'depends-on' on BranchB.
Hence, looking for advice here, in that should 'depends-on' only be used in certain circumstances? ie. when there is a need to run revision from BranchA before BranchB?

Example (if applicable)
The state of my alembic history:

BranchA: (base) revA1 ---> revA2 ---> revA3 ---> revA4 ---> revA5 ---> revA6
                                        \                                \
                                         \ (revB2 depends_on revA3)       \ (revB3 depends_on revA6)
                                          \                                \
BranchB: (base) revB1 -------------------------> revB2 --------------------------> revB3 ---> revB4 (with depends_on=revA6)

When I run 'alembic upgrade heads', I find 'alembic_version' table has only 'revB4'.
Then I add a new revision 'revB5' on top of 'revB4' with SAME depends on BranchA (ie. revA6) :

BranchA: (base) revA1 ---> revA2 ---> revA3 ---> revA4 ---> revA5 ---> revA6
                                        \                                \
                                         \ (revB2 depends_on revA3)       \ (revB3 depends_on revA6)
                                          \                                \
BranchB: (base) revB1 -------------------------> revB2 --------------------------> revB3 ---> revB4 --> revB5 (with depends_on=revA6)

On running 'alembic upgrade heads', the 'alembic_version' table has 'revB5' only.

Now when I run 'alembic downgrade BranchB@revB4 , the alembic_version table has 'revB4' AND 'revA6'. But this is not same as 'before', as it only had 'revB4'.

Now any attempt to upgade (alembic upgrade heads) fails with:

ERROR [alembic.util.messaging] Requested revision revA6 overlaps with other requested revisions revB5
FAILED: Requested revision revA6 overlaps with other requested revisions revB5

The only way I can make this work is manually delete 'revA6', then the upgrade works as per before.

Hence, is this a bug/issue for 'downgrade', as it should not have inserted 'revA6' into the alembic_version table.
OR should revB5 and revB4 be created with no '--depends-on=revA6', as the previous revision already has this dependency.

So next I tested this (alembic 1.5.2), by modifying the revB5 (depends-on to blank). Then I found I can downgrade to revB4 and upgrade to revB5 successfully.

Hence, looking for clarification on when 'depends-on' should be used, when using multiple branches (with separate base)?

@hcpatel hcpatel added the requires triage New issue that requires categorization label Feb 1, 2021
@zzzeek zzzeek added question usage and API questions branch downgrade issues bug Something isn't working versioning model and removed requires triage New issue that requires categorization question usage and API questions labels Feb 2, 2021
@zzzeek
Copy link
Member

zzzeek commented Feb 2, 2021

Describe your question
My use case is 2 branches with their own base to start with, where subsequent revisions from BranchB will depend on BranchA. Each new revision on BranchB I always specify a 'depends on' BranchA (usually --depend_on=branchA@head). But I an finding an issue upgrading to the head of BranchB after I run a downgrade of the latest revision on BranchB (using alembic upgrade heads).
Initially I thought it could be an issue with downgrade, but after some testing realised perhaps it the 'mis-use' of 'depends-on' on BranchB.

OK just so you know, "downgrade" has a lot of not great behaviors with branches, which we'd like to get fixed at some point but there's not resources to attend to them right now. so I would not depend too heavily on downgrades navigating amongst branches perfectly right now.

Hence, looking for advice here, in that should 'depends-on' only be used in certain circumstances? ie. when there is a need to run revision from BranchA before BranchB?

depends-on is used the way it says, when you have a specific revision that "depends on" some revision somewhere else having been run that is not already a direct anscestor of your revision, you can use depends-on to set that up, so that the "depends-on" revision will be upgraded first before the revision in question.

if you dont need your particular revision to run after some particular "depends-on", then I would not use depends on for that, because you don't want to add dependencies between things that aren't necessary, as it leads to a greater chance that the revision tree would have cycles or otherwise become cumbsersome to work with.

Example (if applicable)
The state of my alembic history:

BranchA: (base) revA1 ---> revA2 ---> revA3 ---> revA4 ---> revA5 ---> revA6
                                        \                                \
                                         \ (revB2 depends_on revA3)       \ (revB3 depends_on revA6)
                                          \                                \
BranchB: (base) revB1 -------------------------> revB2 --------------------------> revB3 ---> revB4 (with depends_on=revA6)

When I run 'alembic upgrade heads', I find 'alembic_version' table has only 'revB4'.

So I think that's because revB4 implies revA6 so the version table only stores the revB4 number in this case. Alembic history will show you the actual state, I've replicated your setup here:

$ alembic history -i
511d13b3aade -> 6b2cd82e6e75 (B) (head) (current), revB4
a88b6a4736d1 (0115330fe9d6) -> 511d13b3aade (B), revB3
8896de89b18b (b5299503eb8f) -> a88b6a4736d1 (B), revB2
<base> -> 8896de89b18b (B), revB1
b3a5619c324d -> 0115330fe9d6 (A) (effective head), revA6
492d6df48699 -> b3a5619c324d (A), revA5
b5299503eb8f -> 492d6df48699 (A), revA4
3083374ba2b1 -> b5299503eb8f (A), revA3
a3d012b46845 -> 3083374ba2b1 (A), revA2
<base> -> a3d012b46845 (A), revA1

Then I add a new revision 'revB5' on top of 'revB4' with SAME depends on BranchA (ie. revA6) :

BranchA: (base) revA1 ---> revA2 ---> revA3 ---> revA4 ---> revA5 ---> revA6
                                        \                                \
                                         \ (revB2 depends_on revA3)       \ (revB3 depends_on revA6)
                                          \                                \
BranchB: (base) revB1 -------------------------> revB2 --------------------------> revB3 ---> revB4 --> revB5 (with depends_on=revA6)

On running 'alembic upgrade heads', the 'alembic_version' table has 'revB5' only.

this would be the same idea. it's not possible to have revB5 and not have revA6 implied, so it's not in the table.

Now when I run 'alembic downgrade BranchB@revB4 , the alembic_version table has 'revB4' AND 'revA6'. But this is not same as 'before', as it only had 'revB4'.

yes downgrades are weird, this might be #765.

Now any attempt to upgade (alembic upgrade heads) fails with:

ERROR [alembic.util.messaging] Requested revision revA6 overlaps with other requested revisions revB5
FAILED: Requested revision revA6 overlaps with other requested revisions revB5

yup, it's just a bug, sorry

The only way I can make this work is manually delete 'revA6', then the upgrade works as per before.

Hence, is this a bug/issue for 'downgrade', as it should not have inserted 'revA6' into the alembic_version table.
OR should revB5 and revB4 be created with no '--depends-on=revA6', as the previous revision already has this dependency.

it's most likely just "downgrade", there should not be any way that downgrade puts the version table into a broken state like that.

So next I tested this (alembic 1.5.2), by modifying the revB5 (depends-on to blank). Then I found I can downgrade to revB4 and upgrade to revB5 successfully.

Hence, looking for clarification on when 'depends-on' should be used, when using multiple branches (with separate base)?

It seems clear here the thing that is throwing it off is the double dependency on "A6", so yes I'd try to not use any depends_on that are not strictly necessary, or avoid using fine-grained downgrades.

It's likely this issue would be fixed by #765, or maybe not, so I will keep both of these opened, unfortunately the person who was going to work on the issue seems to have disappeared so once I get some time I may have to open up the revisioning model here again and work a bit with it.

@zzzeek zzzeek changed the title Multiple base and when should depends_on be used downgrade with depedencies leaves revision table in broken state Feb 2, 2021
@zzzeek
Copy link
Member

zzzeek commented Feb 2, 2021

this is actually simple but I dont know how to fix it yet. When you downgrade from B4, it sends "Script(B4, dependencies=A6)" to the migration system that writes to the alembic_version table, which means, "target revision is B4 and remove the A6 dependency". That system assumes the "A6" dependency will no longer apply so A6 becomes a head again. it's not taking into account that A6 is still actually a dependency.

@sqla-tester
Copy link
Collaborator

Mike Bayer has proposed a fix for this issue in the master branch:

Adjust for immediate dependencies that are still ancestors https://gerrit.sqlalchemy.org/c/sqlalchemy/alembic/+/2527

@zzzeek
Copy link
Member

zzzeek commented Feb 2, 2021

OK I think I got this one and it is likely all it's own thing. this will be merged tomorrow and you can test it out, thanks for the report!

@zzzeek
Copy link
Member

zzzeek commented Feb 2, 2021

so I'm ready to put out a point release for this, were you interested in testing to see that it solves your issues? it works on this end. otherwise I will be putting it out today or tomorrow.

@hcpatel
Copy link
Author

hcpatel commented Feb 4, 2021

Apologies, I was on holiday for last 2 days.
I have just downloaded 1.5.4 and can confirm it works.

Many Thanks and Kind Regards,
Hitesh

simonbowly added a commit to simonbowly/alembic that referenced this issue Feb 9, 2021
…g dropped?"

Inclusive flag makes these tests correct.

This reverts commit 3063264.
simonbowly added a commit to simonbowly/alembic that referenced this issue Feb 21, 2021
…g dropped?"

Inclusive flag makes these tests correct.

This reverts commit 3063264.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants