Create better primary keys for subtrees

[fao89]

Author: holger.hees (holger.hees)
Date: 2021-11-12

Redmine Issue: 9566, https://pulp.plan.io/issues/9566

---

Scenario:

My primary pulp instance is hosting 2 distributions of the same repository (like staging and production) which are referencing different versions of the same repository. During my initial run, both distributions are point to version 1. So far so good.

Now I have a secondary pulp instance which is mirroring the 2 primary distributions by creating separate remotes and repositories.

The repository on the primary node contains now a subtree which is identically in version 1 for staging and production. Means it has the same hash.

Now, during the sync process the metadata for this subtree are stored by createing a primary key like "{repodata}-{treeinfo['hash']}". This collides with staging and production, because contentwise and with the hash, the subtree is the same for both staging and production. The key should be something like "{repodata}-{treeinfo['hash']}-{repository_pk}"

#2173

[fao89]

From: pulpbot (pulpbot)
Date: 2021-11-14T10:38:31Z

---

PR: #2176

[fao89]

From: pulpbot (pulpbot)
Date: 2021-11-16T14:57:05Z

---

PR: #2180

[fao89]

From: pulpbot (pulpbot)
Date: 2021-11-30T19:28:53Z

---

PR: #2185

[goosemania]

TL;DR:
After a lot of experiments and investigation of this issue (and fixing other bugs a long the way), I came to a conclusion that we need to change a way we calculate digest for a distribution tree itself or keep track of subtree repo versions.
Changing the naming of subrepos only causes other problems and does not resolve the original problem fully (at least not in my experiments).

I'm using pulp_rpm 3.17.1.

Reproducer:

• sync a repo with a distribution tree and publish/distribute https://fixtures.pulpproject.org/rpm-distribution-tree/
• sync into a new Pulp repo a repo with the same distribution tree but a changed subrepo and publish/distribute https://fixtures.pulpproject.org/rpm-distribution-tree-changed-variant/
• publish/distribute the first repository, observe that its subrepo changed, even though it's the same repository version and no changes were explicitly made for the first repo.

The illustration of the situation below. We do not track repo versions for subrepos, so always the latest version is used.

 ----------------                      ----------------
| repo version 1 |                    | repo version 2 |
 ----------------                      ----------------       
                   \                 /
                      -------------               
                     | dist tree 1 |                       
                      -------------                       
                           |                 
                      ----------                          
                     | sub repo |                       
                      ----------
                        | v1 |
                         ----                             
                        | v2 | 
                         ----                             

Problems with changing subrepos names only:

• we still have only one distribution tree content unit, and the same addons/variants, but we are creating new subrepos. The new subrepos are created but they are not associated with the addon/variant object or distribution tree, they are orphaned. If we start associating them, we'll either orphan old subrepos, or increase the number of subrepos associate with a distribution tree. One way or the other, it's data incorrectness.

Potential solution:

• During sync identify that a subrepo changed, that would mean that a new distribution tree content unit needs to be created and the old one needs to be removed from a repo for which sync is being run.
  • in the first stage before processing the main variant/repo, deal with subrepos: sync/check if they are the same
    • a subrepo name should contain a hash of its repomd.xml (and not a treeinfo hash)
  • if a new distribution tree needs to be created, its digest should be calculated based on its treeinfo digest + all digests from all variants/addons, new addon/variant objects need to be created as well.

@HolgerHees , for you information, since you brought this problem to our attention.
@dralley, @ggainey , @dkliban , @fao89 , any thoughts? any flaws in the suggested solution?

I also believe the fix won't be backportable (3.14/3.16 will need a separate solution). I suggest to encourage folks to upgrade to 3.17 and not even try to come up with a solution for old branches. It would be very error-prone.

[dralley]

I also believe the fix won't be backportable (3.14/3.16 will need a separate solution). I suggest to encourage folks to upgrade to 3.17 and not even try to come up with a solution for old branches. It would be very error-prone.

That's definitely fine for 3.16 since 3.17 is replacing it going forwards, I don't anticipate we will do any more 3.16.z releases.

For 3.14 it would be good to summarize the practical impact of this bug for any users who might be stuck on that release on a BZ. It sounds like it presents a data correctness problem for anyone using the standard test / prod promotion workflow - they might get a newer distribution tree on their slow branch than they were intending.

[fao89]

Potential solution:

• During sync identify that a subrepo changed, that would mean that a new distribution tree content unit needs to be created and the old one needs to be removed from a repo for which sync is being run.

  • in the first stage before processing the main variant/repo, deal with subrepos: sync/check if they are the same

    • a subrepo name should contain a hash of its repomd.xml (and not a treeinfo hash)

  • if a new distribution tree needs to be created, its digest should be calculated based on its treeinfo digest + all digests from all variants/addons, new addon/variant objects need to be created as well.

Good insight! Using repomd.xml hash sounds like the right way to go

[ggainey]

I don't have useful feedback on the solution here - just want to record that pulp/pulpcore#2192 is related, and we'll need to update whatever we/I end up doing to fix that import/export problem, when we address this.

[ggainey]

I'm addressing this under #2278 - closing this as a dup so we can bring all the discussion to one place.