Synching repositories doesn't include binary-all packages

[hstct]

Version
pulpcore: 3.20.0
pulp_deb: 2.19.1

Describe the bug
After synching a debian repository the Packages file in binary-all is empty. This results to those packages not being there at all.

To Reproduce

pulp deb remote create --name bullseye-updates --url https://deb.debian.org/debian/ --policy immediate --distribution bullseye-updates
pulp deb repository create --name bullseye-updates
pulp deb repository sync --name bullseye-updates --remote bullseye-updates
APT_PUBLICATION_HREF=$(pulp deb publication create --repository=bullseye-updates --simple=True --structured=True | jq -r '.pulp_href')
pulp deb distribution create --name=bullseye-updates --base-path=bullseye-updates --publication=${APT_PUBLICATION_HREF}

Check if the file in /pulp/content/bullseye-updates/dists/bullseye-updates/main/binary-all/Packages is not empty.

Expected behavior
The Packages file in binary-all should match the one from the upstream repository and all the packages are also available.

Additional context
It's highly likely that this is caused by the hybrid_format checks in the synchronizing tasks. here and here.

[hstct]

There was another report of this bug. Also using the official debian repository this time however using on_demand and mirror options for the sync and verbatim mode for the publication. Some packages are missing after sync/creating a distribution.

If hybrid_format is forcefully set to False then the packages are present afterwards. Only packages that are only present in the binary-all component are affected by this.

Thanks to @itbane for reporting this issue and also creating a detailed repo to reproduce this issue.

[itbane]

Our initial deployment was pulp-deb 2.16.1, which worked.

I've tested both 2.16.1 (working) and 2.16.2 (not working), so I think we narrowed down the regression to that version.

[quba42]

Just adding this link, since it is quite helpful for testing this: https://github.com/itbane/pulp-sync-issue

[quba42]

Just for completeness sake, it looks like this only affects architecture all packages if the upstream repo uses No-Support-for-Architecture-all: Packages in the Release/InRelease file, AND the package in question is missing from the upstream distributions dists/bullseye/main/binary-all/Packages index, and is instead present only in the architecture specific indices, e.g.: dists/bullseye/main/binary-amd64/Packages.

This situation goes against the spec of the No-Support-for-Architecture-all: Packages format, so this is technically a bug in the official Debian repository, and not a bug in pulp_deb. However, we will treat it as a bug to be handled by pulp_deb, since this pretty much proves no one (including official Debian repos) knows how to adhere to the No-Support-for-Architecture-all: Packages format spec. We will therefore simply forgo the promised performance improvements of this format, in favor of completeness.