Alembic >= 1.6.0 throws "Not a valid downgrade target from current heads".

[msapiro]

Describe the bug

Attempting to test downgrading of all revisions throws alembic.script.revision.RevisionError: Not a valid downgrade target from current heads.

Expected behavior

Downgrading should succeed.

To Reproduce
Please try to provide a Minimal, Complete, and Verifiable example, with the migration script and/or the SQLAlchemy tables or models involved.
See also Reporting Bugs on the website.

This test code which passes with alembic < 1.6.0 fails on 1.6.0 and 1.6.1

import unittest
import alembic.command
from mailman.database.alembic import alembic_cfg

class TestMigrations(unittest.TestCase):
    layer = ConfigLayer

    def setUp(self):
        alembic.command.stamp(alembic_cfg, 'head')

    def test_all_migrations(self):
        script_dir = alembic.script.ScriptDirectory.from_config(alembic_cfg)
        revisions = [sc.revision for sc in script_dir.walk_revisions()]
        with catch_warnings():
            simplefilter('ignore', UserWarning)
            # Alembic/SQLite does not like something about these migrations.
            # They're more or less inconsequential in practice (since users
            # will rarely if ever downgrade their database), but it does
            # clutter up the test output, so just suppress the warning.
            #
            # E.g.
            # alembic/util/messaging.py:69: UserWarning:
            # Skipping unsupported ALTER for creation of implicit constraint
            for revision in revisions:
                alembic.command.downgrade(alembic_cfg, revision)
            revisions.reverse()
            for revision in revisions:
                alembic.command.upgrade(alembic_cfg, revision)

The contents of mailman.database.alembic.init.py are:

from alembic.config import Config
from contextlib import ExitStack
from mailman.utilities.modules import expand_path
from public import public


with ExitStack() as resources:
    cfg_path = expand_path(resources, 'python:mailman.config.alembic')
    public(alembic_cfg=Config(cfg_path))

Error

Traceback (most recent call last):
  File "/var/MM/3/na/mailman/.tox/py37-nocov/lib/python3.7/site-packages/alembic/script/base.py", line 171, in _catch_revision_errors
    yield
  File "/var/MM/3/na/mailman/.tox/py37-nocov/lib/python3.7/site-packages/alembic/script/base.py", line 386, in _downgrade_revs
    for script in revs
  File "/var/MM/3/na/mailman/.tox/py37-nocov/lib/python3.7/site-packages/alembic/script/base.py", line 383, in <listcomp>
    migration.MigrationStep.downgrade_from_script(
  File "/var/MM/3/na/mailman/.tox/py37-nocov/lib/python3.7/site-packages/alembic/script/revision.py", line 711, in iterate_revisions
    assert_relative_length=assert_relative_length,
  File "/var/MM/3/na/mailman/.tox/py37-nocov/lib/python3.7/site-packages/alembic/script/revision.py", line 1212, in _collect_downgrade_revisions
    "Not a valid downgrade target from current heads"
alembic.script.revision.RevisionError: Not a valid downgrade target from current heads

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/MM/3/na/mailman/src/mailman/database/tests/test_migrations.py", line 78, in test_all_migrations
    alembic.command.downgrade(alembic_cfg, revision)
  File "/var/MM/3/na/mailman/.tox/py37-nocov/lib/python3.7/site-packages/alembic/command.py", line 335, in downgrade
    script.run_env()
  File "/var/MM/3/na/mailman/.tox/py37-nocov/lib/python3.7/site-packages/alembic/script/base.py", line 490, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/var/MM/3/na/mailman/.tox/py37-nocov/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
    module = load_module_py(module_id, path)
  File "/var/MM/3/na/mailman/.tox/py37-nocov/lib/python3.7/site-packages/alembic/util/compat.py", line 182, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/MM/3/na/mailman/src/mailman/database/alembic/env.py", line 76, in <module>
    run_migrations_online()
  File "/var/MM/3/na/mailman/src/mailman/database/alembic/env.py", line 70, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/var/MM/3/na/mailman/.tox/py37-nocov/lib/python3.7/site-packages/alembic/runtime/environment.py", line 813, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/var/MM/3/na/mailman/.tox/py37-nocov/lib/python3.7/site-packages/alembic/runtime/migration.py", line 549, in run_migrations
    for step in self._migrations_fn(heads, self):
  File "/var/MM/3/na/mailman/.tox/py37-nocov/lib/python3.7/site-packages/alembic/command.py", line 324, in downgrade
    return script._downgrade_revs(revision, rev)
  File "/var/MM/3/na/mailman/.tox/py37-nocov/lib/python3.7/site-packages/alembic/script/base.py", line 386, in _downgrade_revs
    for script in revs
  File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/var/MM/3/na/mailman/.tox/py37-nocov/lib/python3.7/site-packages/alembic/script/base.py", line 205, in _catch_revision_errors
    compat.raise_(util.CommandError(err.args[0]), from_=err)
  File "/var/MM/3/na/mailman/.tox/py37-nocov/lib/python3.7/site-packages/alembic/util/compat.py", line 294, in raise_
    raise exception
alembic.util.exc.CommandError: Not a valid downgrade target from current heads

Versions.

• OS: Linux
• Python: 3.6 through 3.9
• Alembic: >= 1.6.0
• SQLAlchemy: 1.3.24
• Database: SQlite, MySQL and PostgreSQL
• DBAPI: sqlite, pymysql and psycopg2

Additional context

Have a nice day!

[zzzeek]

hi there -

This looks very similar to #838 just released in 1.6.1 but seems to be a different trace. Like that issue, I can't reproduce here without a self contained example (The code posted seems to be an out of context fragment, for example I dont know what ConfigLayer is also I don't have a mailman environment here). I would need to see the current heads and alembic revision instruction being made - most ideally in the form of "alembic downgrade " from a given head. thanks!

[zzzeek]

@simonbowly do we have a separate codepath when more than one branch is in play? this would then be the same issue as #838 but only caught when there's more than one head, is that it?

[msapiro]

The migrations involved are all at https://gitlab.com/mailman/mailman/-/tree/master/src/mailman/database/alembic/versions

I don't think ConfigLayer is relevant, but it is at https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/testing/layers.py

[simonbowly]

Looks like there's a behaviour change in running alembic downgrade head or equivalent that I think is the cause here (since the code example above runs downgrades of all revisions individually in reverse order).

No revisions are performed in either case, as would be expected, but the new unexpected behaviour is an error rather than just a silent exit.

In <1.6

>> alembic upgrade head
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
>> alembic downgrade head
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.

In >= 1.6

>> alembic upgrade head
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
>> alembic downgrade head
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
ERROR [alembic.util.messaging] Not a valid downgrade target from current heads
  FAILED: Not a valid downgrade target from current heads

[sqla-tester]

Simon Bowly has proposed a fix for this issue in the master branch:

Move empty downgrade revisions check to make it specific to branch filtering case as intended. https://gerrit.sqlalchemy.org/c/sqlalchemy/alembic/+/2811

[msapiro]

I confirm that the patch at https://gerrit.sqlalchemy.org/c/sqlalchemy/alembic/+/2811 fixes my issue. Thank you.

[zzzeek]

1.6.2 is released

[msapiro]

Your responsiveness is amazing. I thought I was good, but you rock! THANK YOU!