Making OpenWISP Upgrades Robust

[pandafy]

While investigating a recent migration issue with dropping jsonfield package, I noticed a gap in our current testing strategy.

Today, our CI primarily validates fresh installations. Every CI run starts from a clean database, applies all migrations, and executes the test suite. While this gives us confidence that a new installation works correctly, it does not fully validate upgrades from previously released versions.

As a result, certain classes of migration issues can go unnoticed until users perform upgrades in production. These issues may involve schema changes, indexes, constraints, migration ordering, data migrations, or assumptions that only hold true on a freshly created database.

Goals

The goal of this proposal is to:

• Continuously validate upgrades from previous releases.
• Detect migration regressions before they reach users.
• Automate the process as much as possible.
• Provide a reusable solution that can be adopted across OpenWISP projects.

Proposed Approach

Introduce automated upgrade fixtures that represent databases created with previous OpenWISP releases.

Whenever a new release is published, a dedicated workflow would automatically:

1. Create a database using the released version.
2. Apply all migrations.
3. Store the resulting database fixture for future CI runs.

Subsequent pull requests could then:

1. Start from one of these fixtures.
2. Apply migrations from the branch under test.
3. Execute the test suite.

This would allow us to continuously verify that upgrades continue to work correctly, rather than only validating fresh installations.

CI Integration

Most OpenWISP repositories already execute a test matrix across multiple Python and Django versions.

Instead of introducing an entirely separate CI pipeline, we could extend one of the existing matrix entries to perform upgrade validation:

1. Start from a fixture generated from a previous release.
2. Configure the test environment to use that fixture.
3. Apply migrations from the branch under test.
4. Run the existing test suite.

This would provide upgrade coverage while keeping CI execution time and complexity under control.

Reusability Across OpenWISP

Since this problem affects multiple projects, it would be useful to implement the fixture generation and upgrade testing logic as reusable GitHub Actions.

Potential adopters include:

• OpenWISP Users
• OpenWISP Controller
• OpenWISP Monitoring
• OpenWISP Notifications
• OpenWISP Radius
• Docker OpenWISP
• Ansible OpenWISP

A shared implementation would reduce duplication and ensure a consistent upgrade-testing strategy across the ecosystem.

Possible Implementations

I have been exploring a few implementation options:

Database Dumps

Generate a database dump whenever a release is published and restore it during CI before running migrations.

Versioned Database Containers

Generate a database fixture during the release process and publish it as a versioned container image. CI could then start directly from that fixture when validating upgrades.

There may be other approaches that achieve the same goal, so I would like to keep the discussion focused on the problem and overall strategy rather than a specific implementation.

Benefits

• Better validation of real upgrade scenarios.
• Earlier detection of migration regressions.
• Increased confidence in releases.
• Reduced risk for production deployments.
• Shared infrastructure that can be reused across OpenWISP projects.

Open Questions

• What is the best way to store and distribute upgrade fixtures?
• Which project would be the best candidate for an initial proof of concept?

I'd be interested in hearing feedback, alternative approaches, or examples of similar solutions used in other projects.

[asmodehn]

I have hit migration issues before that were likely related to the lack of testing around the process of upgrading OpenWISP.
So I also think it is something we are missing in some aspect.

I recently added this test in openwisp-controller as a very limited first step in that direction : https://github.com/openwisp/openwisp-controller/blob/master/openwisp_controller/tests/test_migrationgraph.py
There, I'm leveraging the MigrationLoader and its project_state method to verify that each migration produce valid historical StateApps. Which in effect ensure each migration actually work when applied individually.

With a "narrow focus"/"low-level"

There are things we can already do with some django tooling.
Few packages already exist and are providing various ways to check migrations:

• django-migration-linter
• django-test-migrations
• django-upgrade-check
• django-migration-doctor
• django-safe-migrations

We would need some time to investigate these, learn from other django communities, and come up with a sensible approach...

At a "medium-level"

We could use SQLite, apply the migrations, store the output file somewhere (or a checksum, or extract a schema, or...), and check if the subsequent test runs give the same result.
This is akin to a fixture, and I think there are probably various ways to go about it.
Not sure what is actually feasible or not, and if it is worth the effort required.

Then at a higher-level

We should make sure that workflow like these work as expected, especially for releases:

• v1.1 -> run all tests -> v1.2 upgrade -> run all (upgraded) tests
• v1.2 -> run all tests -> v1.3 upgrade -> run all (upgraded) tests

Not sure if automating it is the way to go, or having having somewhat of a manual process when releases occur, is easier to operate and maintain ?

However, complexity in this area can increase very quickly:

• database testing can be time consuming
• different settings can potentially lead to different DB content, and these may change from one version to the next.
  Django has decorators to manage settings in test, and there is some (limited) tooling around testing migrations.
  I haven't found the django documentation for it, but there is something in the sentry documentation
• these test should also be valid, useful, and extendable for extended apps with potentially different database structure...
• extended apps might depend on previous non-extended apps' database (my use case, admittedly somewhat borderline here)

Current OpenWISP tests are complex, do not always work, and take time to execute.
So we also should make explicit what we will 'not' address in this effort.

Some answers

• What is the best way to store and distribute upgrade fixtures?

I don't think fixture is a workable approach for this... also there would be significant "infrastructure" effort required (creating, storing, moving files/images around, etc.)
I m not entirely sure what is though, so I would follow similar projects (like sentry) for a start...

• Which project would be the best candidate for an initial proof of concept?

In my experience, openwisp-users is simple enough and the data it manages is critical enough, to make it a good candidate for migration testing experiments.

Alternative

Finally I want to side-track a bit, and mention an alternative approach, since in my experience the "upgrading database" strategy has been quite cumbersome.
Instead of :

• v1.1 -> tests -> v1.2 upgrade -> tests

Another possibility, with the same end result, is to:

• v1.1 -> tests -> dump DB -> erase.
• v1.2 -> fresh deploy -> tests -> import DB -> tests

The main benefits are:

• there is no complexity explosion because of various version upgrade path, various database models, various data etc.
• It relies on a 'fresh deployment' process, which we are already testing.
• It relies on a dump/restore process, that is directly testable with a fixture approach and that we should anyway test, to make sure simple backup/restore of data is always working for the user, no matter the version.
  To test this we could store various data dumps of the demo site, and compare with dumps that various tests output...

In that process, the migration testing complexity grows linearly to the number of releases
This is arguably better than the original upgrade process (which I am still using personally), where the migration testing complexity growth is quadratic or worse (depending on what and how we want to test) with the number of releases, for actual database migrations that are already slow.

And I think the community would gain to have a dump/restore workflow documented (Is it pg_dump/pg_restore ? Is it the Export/Import Device buttons on the WebUI?) and actually practiced by everyone, in case of emergency, but also for every deployment, to maximize the chances to learn about it and detect problems as early as possible.

In my mind the 'database upgrade' process is actually an optimization, but we should get the fundamentals (fresh deploy + backup/restore process) right first.

That was long, time to conclude

I didn't expect to write something that long, but here we are. I hope some of it will be a useful piece of information at least.
In the end, we probably need to come up with a clever mix and match of the various aspects discussed above anyway.