[BUG] fails to install from master branch on Debian 11

[dafyddj]

Description of Issue/Question

Problem seems related to #1983

Setup

(Please provide relevant configs (Be sure to remove sensitive info).)

Building a Docker image starting from debian:11

Steps to Reproduce Issue

(Include debug logs if possible, bootstrap-salt.sh -D.)

curl -L https://raw.githubusercontent.com/saltstack/salt-bootstrap/develop/bootstrap-salt.sh | \
sh -s -- -g https://github.com/saltstack/salt.git -XUdfPD -x python3 git master

See https://gitlab.com/dafyddj/salt-image-builder/-/jobs/6084961965

#5 5.502  * ERROR: https://repo.saltproject.io/py3/debian/11/amd64/SALT-PROJECT-GPG-PUBKEY-2023.gpg failed to download to /tmp/salt-gpg-OCqPJnPH.pub
#5 5.502  * ERROR: Failed to run install_debian_git_deps()!!!

INFO: Running version: 2024.01.04
#5 0.240  *  INFO: Executed by: 
#5 0.242  *  INFO: Command line: 'sh -g https://github.com/saltstack/salt.git -XUdfPD -x python3 git master'
#5 0.242  *  WARN: Running the unstable version of bootstrap-salt.sh
#5 0.342 
#5 0.342  *  INFO: System Information:
#5 0.342  *  INFO:   CPU:          AuthenticAMD
#5 0.342  *  INFO:   CPU Arch:     x86_64
#5 0.342  *  INFO:   OS Name:      Linux
#5 0.342  *  INFO:   OS Version:   5.4.109+
#5 0.342  *  INFO:   Distribution: Debian 11
#5 0.342 
#5 0.348  * DEBUG: Binaries will be searched using the following $PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
#5 0.348  *  INFO: Installing minion
#5 0.348  *  INFO: Daemons will not be started
1
#5 0.353  *  WARN: Post Neon git based installations will always install salt
#5 0.353  *  WARN: and its dependencies using pip which will be upgraded to
#5 0.353  *  WARN: at least v9.0.1, and, in case the setuptools version is also
#5 0.353  *  WARN: too old, it will be upgraded to at least v9.1
#5 0.353 
#5 0.353 00  346k  100  346k    0     0  1905k      0 --:--:-- --:--:-- --:--:-- 1905k
#5 0.365  * DEBUG: install_debian_11_git_deps not found....
#5 0.365  * DEBUG: install_debian_11_deps not found....
#5 0.366  *  INFO: Found function install_debian_git_deps
#5 0.366  * DEBUG: DEPS_INSTALL_FUNC=install_debian_git_deps
#5 0.367  * DEBUG: config_debian_11_git_salt not found....
#5 0.368  * DEBUG: config_debian_11_salt not found....
#5 0.368  * DEBUG: config_debian_git_salt not found....
#5 0.369  * DEBUG: config_debian_salt not found....
#5 0.369  *  INFO: Found function config_salt
#5 0.369  * DEBUG: CONFIG_SALT_FUNC=config_salt
#5 0.371  * DEBUG: preseed_debian_11_git_master not found....
#5 0.371  * DEBUG: preseed_debian_11_master not found....
#5 0.371  * DEBUG: preseed_debian_git_master not found....
#5 0.372  * DEBUG: preseed_debian_master not found....
#5 0.372  *  INFO: Found function preseed_master
#5 0.373  * DEBUG: PRESEED_MASTER_FUNC=preseed_master
#5 0.373  * DEBUG: INSTALL_FUNC_NAMES=install_debian_11_git install_debian_11_git install_debian_git
#5 0.374  * DEBUG: install_debian_11_git not found....
#5 0.375  *  INFO: Found function install_debian_git
#5 0.375  * DEBUG: INSTALL_FUNC=install_debian_git
#5 0.376  * DEBUG: install_debian_11_git_post not found....
#5 0.377  * DEBUG: install_debian_11_post not found....
#5 0.377  *  INFO: Found function install_debian_git_post
#5 0.377  * DEBUG: POST_INSTALL_FUNC=install_debian_git_post
#5 0.379  * DEBUG: install_debian_11_git_restart_daemons not found....
#5 0.379  * DEBUG: install_debian_11_restart_daemons not found....
#5 0.380  * DEBUG: install_debian_git_restart_daemons not found....
#5 0.380  *  INFO: Found function install_debian_restart_daemons
#5 0.380  * DEBUG: STARTDAEMONS_INSTALL_FUNC=install_debian_restart_daemons
#5 0.382  * DEBUG: daemons_running_debian_11_git not found....
#5 0.382  * DEBUG: daemons_running_debian_11 not found....
#5 0.383  * DEBUG: daemons_running_debian_git not found....
#5 0.383  * DEBUG: daemons_running_debian not found....
#5 0.384  * DEBUG: daemons_running_git not found....
#5 0.384  *  INFO: Found function daemons_running
#5 0.384  * DEBUG: DAEMONS_RUNNING_FUNC=daemons_running
#5 0.386  * DEBUG: CHECK_SERVICES_FUNC=null
#5 0.386  *  INFO: Running install_debian_git_deps()
#5 0.386  *  WARN: Not starting daemons on Debian based distributions is not working mostly because starting them is the default behaviour.
#5 0.422 Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
#5 0.454 Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
#5 0.468 Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
#5 0.537 Get:4 http://deb.debian.org/debian bullseye/main amd64 Packages [8062 kB]
#5 0.658 Get:5 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [265 kB]
#5 0.739 Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [18.8 kB]
#5 1.691 Fetched 8554 kB in 1s (6620 kB/s)
#5 1.691 Reading package lists...
#5 2.107 Reading package lists...
#5 2.512 Building dependency tree...
#5 2.603 Reading state information...
#5 2.617 Calculating upgrade...
#5 2.705 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
#5 2.714 Reading package lists...
#5 3.118 Building dependency tree...
#5 3.210 Reading state information...
#5 3.301 procps is already the newest version (2:3.3.17-5).
#5 3.301 The following additional packages will be installed:
#5 3.302   libpci3 libyaml-0-2 pci.ids
#5 3.302 Suggested packages:
#5 3.302   bzip2
#5 3.341 The following NEW packages will be installed:
#5 3.344   libpci3 libyaml-0-2 pci.ids pciutils python3-yaml
#5 3.385 0 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
#5 3.385 Need to get 573 kB of archives.
#5 3.385 After this operation, 2284 kB of additional disk space will be used.
#5 3.385 Get:1 http://deb.debian.org/debian bullseye/main amd64 pci.ids all 0.0~2021.02.08-1 [224 kB]
#5 3.432 Get:2 http://deb.debian.org/debian bullseye/main amd64 libpci3 amd64 1:3.7.0-5 [62.7 kB]
#5 3.433 Get:3 http://deb.debian.org/debian bullseye/main amd64 pciutils amd64 1:3.7.0-5 [97.7 kB]
#5 3.435 Get:4 http://deb.debian.org/debian bullseye/main amd64 libyaml-0-2 amd64 0.2.2-1 [49.6 kB]
#5 3.436 Get:5 http://deb.debian.org/debian bullseye/main amd64 python3-yaml amd64 5.3.1-5 [138 kB]
#5 3.577 Fetched 573 kB in 0s (5144 kB/s)
#5 3.596 Selecting previously unselected package pci.ids.
(Reading database ... 14065 files and directories currently installed.)
#5 3.596 Preparing to unpack .../pci.ids_0.0~2021.02.08-1_all.deb ...
#5 3.598 Unpacking pci.ids (0.0~2021.02.08-1) ...
#5 3.650 Selecting previously unselected package libpci3:amd64.
#5 3.650 Preparing to unpack .../libpci3_1%3a3.7.0-5_amd64.deb ...
#5 3.655 Unpacking libpci3:amd64 (1:3.7.0-5) ...
#5 3.[687](https://gitlab.com/dafyddj/salt-image-builder/-/jobs/6084961965#L687) Selecting previously unselected package pciutils.
#5 3.687 Preparing to unpack .../pciutils_1%3a3.7.0-5_amd64.deb ...
#5 3.[692](https://gitlab.com/dafyddj/salt-image-builder/-/jobs/6084961965#L692) Unpacking pciutils (1:3.7.0-5) ...
#5 3.[727](https://gitlab.com/dafyddj/salt-image-builder/-/jobs/6084961965#L727) Selecting previously unselected package libyaml-0-2:amd64.
#5 3.727 Preparing to unpack .../libyaml-0-2_0.2.2-1_amd64.deb ...
#5 3.[730](https://gitlab.com/dafyddj/salt-image-builder/-/jobs/6084961965#L730) Unpacking libyaml-0-2:amd64 (0.2.2-1) ...
#5 3.757 Selecting previously unselected package python3-yaml.
#5 3.759 Preparing to unpack .../python3-yaml_5.3.1-5_amd64.deb ...
#5 3.762 Unpacking python3-yaml (5.3.1-5) ...
#5 3.810 Setting up pci.ids (0.0~2021.02.08-1) ...
#5 3.820 Setting up libyaml-0-2:amd64 (0.2.2-1) ...
#5 3.829 Setting up python3-yaml (5.3.1-5) ...
#5 3.976 Setting up libpci3:amd64 (1:3.7.0-5) ...
#5 3.985 Setting up pciutils (1:3.7.0-5) ...
#5 3.996 Processing triggers for libc-bin (2.31-13+deb11u7) ...
#5 4.030 Reading package lists...
#5 4.441 Building dependency tree...
#5 4.525 Reading state information...
#5 4.636 apt-transport-https is already the newest version (2.2.4).
#5 4.636 ca-certificates is already the newest version (20210119).
#5 4.636 wget is already the newest version (1.21-1+deb11u1).
#5 4.636 The following NEW packages will be installed:
#5 4.636   gnupg2
#5 4.668 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
#5 4.668 Need to get 434 kB of archives.
#5 4.668 After this operation, 453 kB of additional disk space will be used.
#5 4.668 Get:1 http://deb.debian.org/debian bullseye/main amd64 gnupg2 all 2.2.27-2+deb11u2 [434 kB]
#5 4.814 Fetched 434 kB in 0s (6578 kB/s)
#5 4.824 Selecting previously unselected package gnupg2.
(Reading database ... 14126 files and directories currently installed.)
#5 4.831 Preparing to unpack .../gnupg2_2.2.27-2+deb11u2_all.deb ...
#5 4.834 Unpacking gnupg2 (2.2.27-2+deb11u2) ...
#5 4.865 Setting up gnupg2 (2.2.27-2+deb11u2) ...
#5 5.502  * ERROR: https://repo.saltproject.io/py3/debian/11/amd64/SALT-PROJECT-GPG-PUBKEY-2023.gpg failed to download to /tmp/salt-gpg-OCqPJnPH.pub
#5 5.502  * ERROR: Failed to run install_debian_git_deps()!!!
#5 5.502  * DEBUG: Removing the logging pipe /tmp/bootstrap-salt.logpipe
#5 5.503  * DEBUG: Removing the temporary apt error file /tmp/apt_error.1HAQYy
#5 5.507  * DEBUG: Killing logging pipe tee's with pid(s): 21

Versions and Systems

(salt --versions-report, bootstrap-salt.sh -v, system type and version,
cloud/VM provider as appropriate.)

[dmurphy18]

Betting Debian 12 too will have issues, currently doing #1940, and will fix Debian 11 & 12, probably ought to add support for Ubuntu 24.04 given that will be here so soon

[dmurphy18]

closing this in favor of #1940, since it is getting added under that PR, along with massive cleanup

[Kenneth-Sills]

Out of curiosity, why was this closed and merged into #1940? That seems to be a different issue with a much wider scope; this is more of an issue where the two lines of link changes here caused a regression by updating the legacy repository installation codepath with links to the new, incompatible onedir release hierarchy.

• Works:
  • http://repo.saltproject.io/py3/debian/11/amd64/latest/salt-archive-keyring.gpg
  • http://repo.saltproject.io/salt/py3/debian/11/amd64/SALT-PROJECT-GPG-PUBKEY-2023.gpg
• Doesn't Work: http://repo.saltproject.io/py3/debian11/amd64/latest/SALT-PROJECT-GPG-PUBKEY-2023.gpg

Subsequently, the release of v2024.04.03 has broken all Debian 11 VM provisioning at my workplace, since it's now what Vagrant's salt provisioner pulls by default (also Debian 10, looks like... but that's our fault for still being on versions that old 😉).

Was surprised to see those novel failed tests were just kind of ignored... Furthermore, the PR itself also changed code that doesn't seem like it could have even been the culprit of the quoted issue, since __install_saltstack_debian_repository was updated instead of __install_saltstack_debian_onedir_repository, and the former has no ability to generate the problematic https://repo.saltproject.io/salt/py3/debian/12/amd64/latest/salt-archive-keyring.gpg link without a custom repo URL being passed in with -R.

What seems more likely is that the latter function attempts to download from legacy before trying the onedir repository. The PR submitter never actually demonstrated anywhere that their bootstrapping failed. In fact, right after the error line in their terminal we see "Hit: ...", indicating that execution continued into the fallback where the SALT-PROJECT-GPG-PUBKEY-2023.gpg filename is tried instead. And looking at this issue we can see exactly that, resulting in a successful bootstrapping. The only "issue" here is that the script should more clearly explain that it's a soft failure and it's moving on to a different method OR send that initial command's STDERR to the void.

Would it be possible to re-open this ticket and work on an expedited fix? I'd be happy to submit a PR if it would speed things up. It seems like the original PR just needs to be reverted, full-stop. If my hunch above is correct, it caused a regression and fixed nothing. Let me know if I'm sorely mistaken.

Sorry for the long post, @dmurphy18, I hope you enjoy your upcoming vacation! From that huge PR I can see you certainly deserve it! 😆

EDIT: Submitted a revert PR!

[dmurphy18]

@dafyddj Will be across the Irish Sea, so soon, Dublin, bye Utah - thanks for the wishes on vacation. The problem here is

Doesn't Work: http://repo.saltproject.io/py3/debian11/amd64/latest/SALT-PROJECT-GPG-PUBKEY-2023.gpg

http://repo.saltproject.io/py3 implies classic packaging which uses the SHA-1 key, and not the SALT-PROJECT-GPG-PUBKEY-2023.gpg SHA-256 key.

Bootstrap, once I dug into it, was a rats-nest and mess, hence the large amount of re-work and removal of kitchen-salt (why ruby based stuff with a python house :( ), Kitchen-salt is no longer supported (at least by Salt team), was a pet project from a previous team member back in 2017.

So closed, because wrong and classic packaging is not longer supported, and massive cleanup being done.