downgrade with depedencies leaves revision table in broken state

[hcpatel]

Describe your question
My use case is 2 branches with their own base to start with, where subsequent revisions from BranchB will depend on BranchA. Each new revision on BranchB I always specify a 'depends on' BranchA (usually --depend_on=branchA@head). But I an finding an issue upgrading to the head of BranchB after I run a downgrade of the latest revision on BranchB (using alembic upgrade heads).
Initially I thought it could be an issue with downgrade, but after some testing realised perhaps it the 'mis-use' of 'depends-on' on BranchB.
Hence, looking for advice here, in that should 'depends-on' only be used in certain circumstances? ie. when there is a need to run revision from BranchA before BranchB?

Example (if applicable)
The state of my alembic history:

BranchA: (base) revA1 ---> revA2 ---> revA3 ---> revA4 ---> revA5 ---> revA6
                                        \                                \
                                         \ (revB2 depends_on revA3)       \ (revB3 depends_on revA6)
                                          \                                \
BranchB: (base) revB1 -------------------------> revB2 --------------------------> revB3 ---> revB4 (with depends_on=revA6)

When I run 'alembic upgrade heads', I find 'alembic_version' table has only 'revB4'.
Then I add a new revision 'revB5' on top of 'revB4' with SAME depends on BranchA (ie. revA6) :

BranchA: (base) revA1 ---> revA2 ---> revA3 ---> revA4 ---> revA5 ---> revA6
                                        \                                \
                                         \ (revB2 depends_on revA3)       \ (revB3 depends_on revA6)
                                          \                                \
BranchB: (base) revB1 -------------------------> revB2 --------------------------> revB3 ---> revB4 --> revB5 (with depends_on=revA6)

On running 'alembic upgrade heads', the 'alembic_version' table has 'revB5' only.

Now when I run 'alembic downgrade BranchB@revB4 , the alembic_version table has 'revB4' AND 'revA6'. But this is not same as 'before', as it only had 'revB4'.

Now any attempt to upgade (alembic upgrade heads) fails with:

ERROR [alembic.util.messaging] Requested revision revA6 overlaps with other requested revisions revB5
FAILED: Requested revision revA6 overlaps with other requested revisions revB5

The only way I can make this work is manually delete 'revA6', then the upgrade works as per before.

Hence, is this a bug/issue for 'downgrade', as it should not have inserted 'revA6' into the alembic_version table.
OR should revB5 and revB4 be created with no '--depends-on=revA6', as the previous revision already has this dependency.

So next I tested this (alembic 1.5.2), by modifying the revB5 (depends-on to blank). Then I found I can downgrade to revB4 and upgrade to revB5 successfully.

Hence, looking for clarification on when 'depends-on' should be used, when using multiple branches (with separate base)?

[zzzeek]

Describe your question
My use case is 2 branches with their own base to start with, where subsequent revisions from BranchB will depend on BranchA. Each new revision on BranchB I always specify a 'depends on' BranchA (usually --depend_on=branchA@head). But I an finding an issue upgrading to the head of BranchB after I run a downgrade of the latest revision on BranchB (using alembic upgrade heads).
Initially I thought it could be an issue with downgrade, but after some testing realised perhaps it the 'mis-use' of 'depends-on' on BranchB.

OK just so you know, "downgrade" has a lot of not great behaviors with branches, which we'd like to get fixed at some point but there's not resources to attend to them right now. so I would not depend too heavily on downgrades navigating amongst branches perfectly right now.

Hence, looking for advice here, in that should 'depends-on' only be used in certain circumstances? ie. when there is a need to run revision from BranchA before BranchB?

depends-on is used the way it says, when you have a specific revision that "depends on" some revision somewhere else having been run that is not already a direct anscestor of your revision, you can use depends-on to set that up, so that the "depends-on" revision will be upgraded first before the revision in question.

if you dont need your particular revision to run after some particular "depends-on", then I would not use depends on for that, because you don't want to add dependencies between things that aren't necessary, as it leads to a greater chance that the revision tree would have cycles or otherwise become cumbsersome to work with.

Example (if applicable)
The state of my alembic history:

BranchA: (base) revA1 ---> revA2 ---> revA3 ---> revA4 ---> revA5 ---> revA6
                                        \                                \
                                         \ (revB2 depends_on revA3)       \ (revB3 depends_on revA6)
                                          \                                \
BranchB: (base) revB1 -------------------------> revB2 --------------------------> revB3 ---> revB4 (with depends_on=revA6)

When I run 'alembic upgrade heads', I find 'alembic_version' table has only 'revB4'.

So I think that's because revB4 implies revA6 so the version table only stores the revB4 number in this case. Alembic history will show you the actual state, I've replicated your setup here:

$ alembic history -i
511d13b3aade -> 6b2cd82e6e75 (B) (head) (current), revB4
a88b6a4736d1 (0115330fe9d6) -> 511d13b3aade (B), revB3
8896de89b18b (b5299503eb8f) -> a88b6a4736d1 (B), revB2
<base> -> 8896de89b18b (B), revB1
b3a5619c324d -> 0115330fe9d6 (A) (effective head), revA6
492d6df48699 -> b3a5619c324d (A), revA5
b5299503eb8f -> 492d6df48699 (A), revA4
3083374ba2b1 -> b5299503eb8f (A), revA3
a3d012b46845 -> 3083374ba2b1 (A), revA2
<base> -> a3d012b46845 (A), revA1

Then I add a new revision 'revB5' on top of 'revB4' with SAME depends on BranchA (ie. revA6) :

BranchA: (base) revA1 ---> revA2 ---> revA3 ---> revA4 ---> revA5 ---> revA6
                                        \                                \
                                         \ (revB2 depends_on revA3)       \ (revB3 depends_on revA6)
                                          \                                \
BranchB: (base) revB1 -------------------------> revB2 --------------------------> revB3 ---> revB4 --> revB5 (with depends_on=revA6)

On running 'alembic upgrade heads', the 'alembic_version' table has 'revB5' only.

this would be the same idea. it's not possible to have revB5 and not have revA6 implied, so it's not in the table.

Now when I run 'alembic downgrade BranchB@revB4 , the alembic_version table has 'revB4' AND 'revA6'. But this is not same as 'before', as it only had 'revB4'.

yes downgrades are weird, this might be #765.

Now any attempt to upgade (alembic upgrade heads) fails with:

ERROR [alembic.util.messaging] Requested revision revA6 overlaps with other requested revisions revB5
FAILED: Requested revision revA6 overlaps with other requested revisions revB5

yup, it's just a bug, sorry

The only way I can make this work is manually delete 'revA6', then the upgrade works as per before.

Hence, is this a bug/issue for 'downgrade', as it should not have inserted 'revA6' into the alembic_version table.
OR should revB5 and revB4 be created with no '--depends-on=revA6', as the previous revision already has this dependency.

it's most likely just "downgrade", there should not be any way that downgrade puts the version table into a broken state like that.

So next I tested this (alembic 1.5.2), by modifying the revB5 (depends-on to blank). Then I found I can downgrade to revB4 and upgrade to revB5 successfully.

Hence, looking for clarification on when 'depends-on' should be used, when using multiple branches (with separate base)?

It seems clear here the thing that is throwing it off is the double dependency on "A6", so yes I'd try to not use any depends_on that are not strictly necessary, or avoid using fine-grained downgrades.

It's likely this issue would be fixed by #765, or maybe not, so I will keep both of these opened, unfortunately the person who was going to work on the issue seems to have disappeared so once I get some time I may have to open up the revisioning model here again and work a bit with it.

[zzzeek]

this is actually simple but I dont know how to fix it yet. When you downgrade from B4, it sends "Script(B4, dependencies=A6)" to the migration system that writes to the alembic_version table, which means, "target revision is B4 and remove the A6 dependency". That system assumes the "A6" dependency will no longer apply so A6 becomes a head again. it's not taking into account that A6 is still actually a dependency.

[sqla-tester]

Mike Bayer has proposed a fix for this issue in the master branch:

Adjust for immediate dependencies that are still ancestors https://gerrit.sqlalchemy.org/c/sqlalchemy/alembic/+/2527

[zzzeek]

OK I think I got this one and it is likely all it's own thing. this will be merged tomorrow and you can test it out, thanks for the report!

[zzzeek]

so I'm ready to put out a point release for this, were you interested in testing to see that it solves your issues? it works on this end. otherwise I will be putting it out today or tomorrow.