migrations: Update backfill assumptions

[efritz]

Fixes the root cause of #55414, which broke upgrades between v5.0[3,6] and v5.1.5. We will need to provide instructions for users that did hit v5.1.5, as they will have already experienced the effect of this bug. They can simply run the missing migrations we can supply as a support packet. This should fix the issue from occurring in v5.1.6 and later.

The issue came from v5.1.5 incorrectly backfilling a set of migrations. When we initialize the database, we insert "backfilled" migration records for migrations that existed in the past but are no longer in-tree (after being squashed away). The logic for this assumed that version branches wouldn't redefine migration ancestry (we still shouldn't, but we can fix the effect of there).

Since we've started backporting Cody features, we've introduced at least one occurrence where the ancestry of a migration definition changes between the 5.0 and main branches. In these cases, a migration was backported without all of its ancestors. This creates a situation where moving from 5.0 to 5.1 branches a migration suddenly gets an (unapplied) parent added to it.

This breaks the assumption that any migration successfully applied has also had its ancestors applied. But sometimes Fry has to go back in time and become his own grandfather. 🤷

This PR fixes the issue by instead of backfilling _all ancestors of all applied migrations-, we only backfill the (get read for this, it's a mouth-full): all ancestors of the root of a version of which all leaf migrations have been applied. :clueless:

This is a less eager way to backfill, as we will only backfill definitions prior to a completely applied set of migrations for some particular version.

Test plan

Updated unit tests. Will backport these changes to a local 5.1 branch and test the v5.0.6->v5.1.5(+ this fix) upgrade to see if we still have issues.

[coury-clark]

all ancestors of the root of a version of which all leaf migrations have been applied

😅

[efritz]

all ancestors of the root of a version of which all leaf migrations have been applied

😅

Correct response! Maybe this is helpful?

[coury-clark]

@efritz very! I think this should go in the docs somewhere, so when we inevitably find something else we need to change we don't have to relearn how this all works 😄.

[coury-clark]

Why would this ever be negative?

[efritz]

Squashed migrations become a pair of [-ID, ID] migrations where the negative one has privileged commands (extension installation). We have to do this because there's no known gaps between IDs when they were sequential.

[efritz]

This basically says "if it's the extension one, just go to its twin"

[efritz]

I think this should go in the docs somewhere, so when we inevitably find something else we need to change we don't have to relearn how this all works

@DaedalusG would you be game to make some internal doc pages (even if blank) to start filling out? I'm happy to write or brain dump about any of the internals, but the wide canvas is too daunting to start off on my own with.

[DaedalusG]

@efritz @coury-clark I was actually thinking the same thing. As an exercise for knowledge transfer going through and writing comprehensive docs about migrator upgrades and the dependencies on the release docs is in order 👍, it'll be a great way to deeply learn this part of the codebase too 😎

[DaedalusG]

I was actually planning on doing something like this with advisement as is 👍