Copy alembic files to scaffolds

[koenedaele]

Let's add the alembic files to scaffolds as well. That way, an implementation can maintain it's own alembic migrations. Whenever an Atramhasis update is scaffolded in, the user should create a new merge? Not sure if this will work cleanly.

[koenedaele]

@cedrikv We should see how we want to handle this or not. Think you solved some things for the last version of thesaurus. So, maybe this ticket is no longer necessary.

[cedrikv]

No proper solution yet, I just created new alembic versions in thesaurus with the updates from atramhasis.

[koenedaele]

@Wim-De-Clercq Is this something you can look into? The basic problem is that we have alembic migrations in our base package (atramhasis), but that we sometimes want to add migrations to our implementation package (thesaurus in our case) as well, eg. when we add a new conceptscheme. In this case, when we update to a new version of Atramhasis it has no knowledge of the alembic revisions that were created in the local package. Currently, we're just copy-pasting new alembic migrations and doing manual fixes. But is there a better way? Using alemic branches in some way (https://alembic.sqlalchemy.org/en/latest/branches.html)? Or something completely different?

[Wim-De-Clercq]

I do indeed think the branches could be a solution, yes. I'll have to test it a bit to be sure though.

[Wim-De-Clercq]

I did some testing and the following works:

• delete all atramhasis revision copies from thesaurus. What's left:

- In thesaurus I made the first revision look like

revision = '5cfd7365fc93'
down_revision = None
depends_on = '3bcf11802900'  # this is a revision  in atramhasis project

branch_labels = ('thesaurus',)

I also added an atramhasis label in atramhasis project.

Then in the alembic.ini I added

version_locations = %(here)s/alembic/versions atramhasis:alembic/versions

And then setting everything up looks like:

Everything works fine as of yet, but I do still have an open question that I can't find a solution for, so I asked it in the alembic repo.
sqlalchemy/alembic#725

When that is resolved, all that's left is I must make sure that an upgrade from the current installed revision to this new revision setup works flawlessly for dev/test/prod servers.

[Wim-De-Clercq]

The following (little hacky) code can be used to add a custom depends_on to add a dependency within atramhasis revisions:

with context.begin_transaction():
    revision_map = context.script.revision_map
    revision = revision_map._revision_map['b04fd493106b']
    revision.dependencies = '5cfd7365fc93'
    revision_map.add_revision(revision, _replace=True)
    context.run_migrations()

This is code in alembic/env.py in the thesaurus project.

We don't need it now, but I write it here so that hopefully when someone in the future needs they may see it.

[Wim-De-Clercq]

Right now, thesaurus copies all new revisoins from atramhasis over to its own alembic (manual action)

Ideally, I would like to change it so that in alembic.ini we configure this:

version_locations = %(here)s/alembic/versions atramhasis:alembic/versions

Then all revisions from atramhasis will be automatically imported and processed.
Projects would then have 2 alembic branches:

atramhasis: ahash1 > ahash2 > ahash3 > ahash4
thesaurus: thash1 > thash2

There would also be 2 heads present in the alembic_revisions table in the DB. This table contains 2 rows: ahash4 and thash2

It is, imo, by far the cleanest project setup. When atramhasis updates, all new revisions are automatically seen and executed.
But the problem is the current existence of projects like thesaurus or any other project using atramhasis already.
How do we change the current alembic setup to the new one nicely.

I can clean up the revisions in thesaurus, and manually insert a row in the alembic_version table.
See before and after of the alembic_version

thesaurus=# table alembic_version ;
 version_num  
--------------
 492c4d0b6dc2
 
 thesaurus=# table alembic_version ;
 version_num  
--------------
 184f1bbcb916
 492c4d0b6dc2

184f1bbcb916 is the latest revision within atramhasis. 492c4d0b6dc2 is the latest revision within thesaurus.

But what about other projects in the world that use atramhasis. We can't just ask them to do a change like this themselves, can we?

[cedrikv]

@Wim-De-Clercq We are missing an estimate for this one