Handle upstream package indices containing packages with the wrong architecture

[quba42]

Minimal reproducer:

pip install pulp-cli-deb

# Constants:
ENTITIES_NAME='test'
REMOTE_OPTIONS='--url=https://apt.releases.hashicorp.com --distribution=bionic --component=main --architecture=amd64 --policy=on_demand'

# Steps using CLI:
pulp deb remote create --name=${ENTITIES_NAME} ${REMOTE_OPTIONS}
pulp deb repository create --name=${ENTITIES_NAME} --remote=${ENTITIES_NAME}
pulp deb repository sync --name=${ENTITIES_NAME}
APT_PUBLICATION_HREF=$(pulp deb publication create --repository=${ENTITIES_NAME} --simple=True --structured=True | jq -r '.pulp_href')
# Creating the publication fails!
pulp deb distribution create --name=${ENTITIES_NAME} --base-path=${ENTITIES_NAME} --publication=${APT_PUBLICATION_HREF}

Error message:

    "error"=>
     {"traceback"=>
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 317, in _perform_task\n" +
       "    result = func(*args, **kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/publishing.py\", line 168, in publish\n" +
       "    prc.package\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/publishing.py\", line 207, in add_package\n" +
       "    self.package_index_files[package.architecture][0]\n",
      "description"=>"'arm64'"},

[quba42]

Analysis revealed that for example the HashiCorp repo package index dists/bionic/main/binary-amd64/Packages has package paragraphs for all kinds of architectures that are not amd64 in it! E.g.:

Package: consul-terraform-sync-enterprise
Version: 0.4.0+ent
Section:
Priority:
Architecture: i386
Maintainer: HashiCorp
Installed-Size: 30802
Depends: openssl
Homepage: https://github.com/hashicorp/consul-terraform-sync
Description: Consul Terraform Sync is a service-oriented tool for managing network infrastructure near real-time.
Filename: pool/amd64/main/consul-terraform-sync-enterprise_0.4.0+ent_i386.deb
SHA1: a0ee7bf6deec3bb36fc0fae523a73f29baac5ce3
SHA256: 3c513be065c0c60f879602c3e25752ef8594f22c804be13101cff83244615a48
Size: 15813840

This is mainly a bug with the upstream repo (which is structured in a number of strange/specification and good practice defying ways), but we can nevertheless mitigate such cases by skipping the relevant packages during sync.

[evgeni]

Do you know what HCP uses for managing that repo? Might be worth escalating there?

(A more resilient pulp-deb ia still a good thing)

[quba42]

Do you know what HCP uses for managing that repo? Might be worth escalating there?

I do not know where I might report the upstream issue, though it would be good to do so. The number of oddities in that repo makes me think it might be something they built themselves... Perhaps just email them?

[quba42]

Note: I alerted HashiCorp to our findings via the from found at: https://support.hashicorp.com/hc/en-us/requests/new

[quba42]

Apparently the form I used is for paying enterprise customers only, so I also opened a thread in the HashiCorp community forum: https://discuss.hashicorp.com/t/buggy-weirdly-structured-hashicorp-apt-repo/34814