Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bootstrap of Debian family broken in 2017.08.17 #1137

Closed
EvaSDK opened this issue Aug 23, 2017 · 9 comments · Fixed by #1139
Closed

Bootstrap of Debian family broken in 2017.08.17 #1137

EvaSDK opened this issue Aug 23, 2017 · 9 comments · Fixed by #1139
Labels
Bug stale
Milestone
Approved

Comments

@EvaSDK
Copy link

EvaSDK commented Aug 23, 2017

Description of Issue/Question

Setup

I run minions in a network with Internet access only available through an HTTP proxy.
After upgrading to 2017.08.17, boostrapping new minions fails due to changes introduced in commit 0e45ba1.

Logs

[...]
Dépaquetage de gnupg1-curl (1.4.21-4) ...
Sélection du paquet gnupg1-l10n précédemment désélectionné.
Préparation du dépaquetage de .../gnupg1-l10n_1.4.21-4_all.deb ...
Dépaquetage de gnupg1-l10n (1.4.21-4) ...
Paramétrage de gnupg1 (1.4.21-4) ...
Paramétrage de gnupg1-l10n (1.4.21-4) ...
Traitement des actions différées (« triggers ») pour man-db (2.7.6.1-2) ...
Paramétrage de gnupg1-curl (1.4.21-4) ...
Warning: apt-key output should not be parsed (stdout is not a terminal)
Executing: /tmp/apt-key-gpghome.tlIJzNCn0a/gpg.1.sh --keyserver-options ca-cert-file=/etc/ssl/certs/ca-certificates.crt --fetch-keys https://repo.saltstack.com/apt/debian/9/amd64/2016.3/SALTSTACK-GPG-KEY.pub
gpg: keyserver option 'ca-cert-file' is obsolete; please use 'hkp-cacert' in dirmngr.conf
gpg: requesting key from 'https://repo.saltstack.com/apt/debian/9/amd64/2016.3/SALTSTACK-GPG-KEY.pub'
gpg: WARNING: unable to fetch URI https://repo.saltstack.com/apt/debian/9/amd64/2016.3/SALTSTACK-GPG-KEY.pub: Connexion terminée par expiration du délai d'attente
Ign:1 http://ftp.fr.debian.org/debian stretch InRelease
Atteint:2 http://ftp.fr.debian.org/debian stretch-updates InRelease
Atteint:3 http://security.debian.org/debian-security stretch/updates InRelease
Atteint:4 http://ftp.fr.debian.org/debian stretch Release
Réception de:6 https://repo.saltstack.com/apt/debian/9/amd64/2016.3 stretch InRelease [2 841 B]
Ign:6 https://repo.saltstack.com/apt/debian/9/amd64/2016.3 stretch InRelease
Réception de:7 https://repo.saltstack.com/apt/debian/9/amd64/2016.3 stretch/main amd64 Packages [4 464 B]
7 305 o réceptionnés en 1s (6 567 o/s)
Lecture des listes de paquets…
W: Erreur de GPG : https://repo.saltstack.com/apt/debian/9/amd64/2016.3 stretch InRelease : Les signatures suivantes n'ont pas pu être vérifiées car la clé publique n'est pas disponible : NO_PUBKEY 0E08A149DE57BFBE
W: The repository 'https://repo.saltstack.com/apt/debian/9/amd64/2016.3 stretch InRelease' is not signed.
[...]

You can see apt-key timing out and later apt failing due to security warning.

Versions and Systems

# salt --versions-report
Salt Version:
           Salt: 2016.11.7
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Jan 19 2017, 14:48:08)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1
 
System Versions:
           dist: debian 9.1 
        machine: x86_64
        release: 4.9.0-3-amd64
         system: Linux
        version: debian 9.1 

# /srv/salt-bootstrap/bootstrap-salt.sh -v
/srv/salt-bootstrap/bootstrap-salt.sh -- Version 2017.08.17
@EvaSDK
Copy link
Author

EvaSDK commented Aug 24, 2017

Confirming reverting salt-bootstrap to v2017.05.24 fixed the problem.

@vutny
Copy link
Contributor

vutny commented Aug 25, 2017 •
edited

It should work in Debian 8 "jessie", because there are old version of GnuPG which uses curl backend to retrieve the key and it honors http_proxy envrionment variable.

However, in GnuPG 2.1 (introduced in Debian 9 "stretch" and Ubuntu starting from 16.10 I believe) the key is going to be downloaded with dirmngr, which ignores http_proxy variable by default.

Thanks for reporting, this definitely should be fixed and implemented reliably. @rallytime I'll try to get on this next week or so.

@rallytime
Copy link
Contributor

@EvaSDK Thanks for reporting this issue.

@vutny Sounds great! I look forward to seeing your fix.

@rallytime rallytime added this to the Approved milestone Aug 28, 2017
@vutny vutny mentioned this issue Aug 30, 2017
Merged
@vutny
Copy link
Contributor

vutny commented Aug 30, 2017

@EvaSDK I've attempted to fix this in PR #1139. Please give it a try.

If your proxy configuration does not allow transparent TLS connections (without sending HTTP CONNECT), you might consider passing -l option to the bootstrap script.

rallytime pushed a commit that referenced this issue Sep 7, 2017
Merge pull request #1139 from vutny/fix-1137
f24508b 
Fix #1137: import GPG key through an HTTP(S) proxy
@EvaSDK
Copy link
Author

EvaSDK commented Oct 9, 2017

Finally had time to check this today with current develop HEAD:

# git describe --tags
v2014.07.29-1164-g53f6d68

It still fails to retrieve the key. Maybe it needs an explicit HTTPS proxy setting.
Here are the logs:

# salt-cloud -ym /etc/salt/cloud.maps.d/saltify.conf canalths01.prd.lexfo.fr
Warning: Permanently added '10.105.4.172' (ECDSA) to the list of known hosts.
lundi 9 octobre 2017, 13:03:07 (UTC+0200)
Warning: Permanently added '10.105.4.172' (ECDSA) to the list of known hosts.
Connection to 10.105.4.172 closed.
Warning: Permanently added '10.105.4.172' (ECDSA) to the list of known hosts.
Connection to 10.105.4.172 closed.
Warning: Permanently added '10.105.4.172' (ECDSA) to the list of known hosts.
Connected to 10.105.4.172.
sftp> put  /tmp/tmpKQwHoI /tmp/.saltcloud-a2e37c35-ff2f-4bd4-a15a-d01afc19be2c/minion.pem
Uploading /tmp/tmpKQwHoI to /tmp/.saltcloud-a2e37c35-ff2f-4bd4-a15a-d01afc19be2c/minion.pem
/tmp/tmpKQwHoI                                                         100% 3246    15.0MB/s   00:00    
Warning: Permanently added '10.105.4.172' (ECDSA) to the list of known hosts.
Connection to 10.105.4.172 closed.
Warning: Permanently added '10.105.4.172' (ECDSA) to the list of known hosts.
Connected to 10.105.4.172.
sftp> put  /tmp/tmpaGF9TV /tmp/.saltcloud-a2e37c35-ff2f-4bd4-a15a-d01afc19be2c/minion.pub
Uploading /tmp/tmpaGF9TV to /tmp/.saltcloud-a2e37c35-ff2f-4bd4-a15a-d01afc19be2c/minion.pub
/tmp/tmpaGF9TV                                                         100%  799     5.5MB/s   00:00    
Warning: Permanently added '10.105.4.172' (ECDSA) to the list of known hosts.
Connected to 10.105.4.172.
sftp> put  /tmp/tmpDPmM6q /tmp/.saltcloud-a2e37c35-ff2f-4bd4-a15a-d01afc19be2c/minion
Uploading /tmp/tmpDPmM6q to /tmp/.saltcloud-a2e37c35-ff2f-4bd4-a15a-d01afc19be2c/minion
/tmp/tmpDPmM6q                                                         100%   81   495.4KB/s   00:00    
Warning: Permanently added '10.105.4.172' (ECDSA) to the list of known hosts.
Connected to 10.105.4.172.
sftp> put  /tmp/tmpRkTbsc /tmp/.saltcloud-a2e37c35-ff2f-4bd4-a15a-d01afc19be2c/deploy.sh
Uploading /tmp/tmpRkTbsc to /tmp/.saltcloud-a2e37c35-ff2f-4bd4-a15a-d01afc19be2c/deploy.sh
/tmp/tmpRkTbsc                                                         100%  244KB  81.8MB/s   00:00    
Warning: Permanently added '10.105.4.172' (ECDSA) to the list of known hosts.
Connection to 10.105.4.172 closed.
Warning: Permanently added '10.105.4.172' (ECDSA) to the list of known hosts.
 *  INFO: Running version: 2017.08.17
 *  INFO: Executed by: 
 *  INFO: Command line: '/tmp/.saltcloud-a2e37c35-ff2f-4bd4-a15a-d01afc19be2c/deploy.sh -c /tmp/.saltcloud-a2e37c35-ff2f-4bd4-a15a-d01afc19be2c -D -F -H http://10.105.4.1:3128/ stable 2016.3'
 *  WARN: Running the unstable version of bootstrap-salt.sh

 *  INFO: System Information:
 *  INFO:   CPU:          GenuineIntel
 *  INFO:   CPU Arch:     x86_64
 *  INFO:   OS Name:      Linux
 *  INFO:   OS Version:   4.9.0-3-amd64
 *  INFO:   Distribution: Debian 9.1

 * DEBUG: Binaries will be searched using the following $PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 *  INFO: Using http proxy http://10.105.4.1:3128/
 *  INFO: Installing minion
 * DEBUG: install_debian_9_stable_deps not found....
 * DEBUG: install_debian_9_1_stable_deps not found....
 * DEBUG: install_debian_9_deps not found....
 * DEBUG: install_debian_9_1_deps not found....
 * DEBUG: install_debian_stable_deps not found....
 *  INFO: Found function install_debian_deps
 * DEBUG: DEPS_INSTALL_FUNC=install_debian_deps
 * DEBUG: config_debian_9_stable_salt not found....
 * DEBUG: config_debian_9_1_stable_salt not found....
 * DEBUG: config_debian_9_salt not found....
 * DEBUG: config_debian_9_1_salt not found....
 * DEBUG: config_debian_stable_salt not found....
 * DEBUG: config_debian_salt not found....
 *  INFO: Found function config_salt
 * DEBUG: CONFIG_SALT_FUNC=config_salt
 * DEBUG: preseed_debian_9_stable_master not found....
 * DEBUG: preseed_debian_9_1_stable_master not found....
 * DEBUG: preseed_debian_9_master not found....
 * DEBUG: preseed_debian_9_1_master not found....
 * DEBUG: preseed_debian_stable_master not found....
 * DEBUG: preseed_debian_master not found....
 *  INFO: Found function preseed_master
 * DEBUG: PRESEED_MASTER_FUNC=preseed_master
 *  INFO: Found function install_debian_9_stable
 * DEBUG: INSTALL_FUNC=install_debian_9_stable
 * DEBUG: install_debian_9_stable_post not found....
 * DEBUG: install_debian_9_1_stable_post not found....
 * DEBUG: install_debian_9_post not found....
 * DEBUG: install_debian_9_1_post not found....
 * DEBUG: install_debian_stable_post not found....
 * DEBUG: install_debian_post not found....
 * DEBUG: POST_INSTALL_FUNC=null
 * DEBUG: install_debian_9_stable_restart_daemons not found....
 * DEBUG: install_debian_9_1_stable_restart_daemons not found....
 * DEBUG: install_debian_9_restart_daemons not found....
 * DEBUG: install_debian_9_1_restart_daemons not found....
 * DEBUG: install_debian_stable_restart_daemons not found....
 *  INFO: Found function install_debian_restart_daemons
 * DEBUG: STARTDAEMONS_INSTALL_FUNC=install_debian_restart_daemons
 * DEBUG: daemons_running_debian_9_stable not found....
 * DEBUG: daemons_running_debian_9_1_stable not found....
 * DEBUG: daemons_running_debian_9 not found....
 * DEBUG: daemons_running_debian_9_1 not found....
 * DEBUG: daemons_running_debian_stable not found....
 * DEBUG: daemons_running_debian not found....
 *  INFO: Found function daemons_running
 * DEBUG: DAEMONS_RUNNING_FUNC=daemons_running
 * DEBUG: install_debian_9_stable_check_services not found....
 * DEBUG: install_debian_9_1_stable_check_services not found....
 * DEBUG: install_debian_9_check_services not found....
 * DEBUG: install_debian_9_1_check_services not found....
 * DEBUG: install_debian_stable_check_services not found....
 *  INFO: Found function install_debian_check_services
 * DEBUG: CHECK_SERVICES_FUNC=install_debian_check_services
 *  INFO: Running install_debian_deps()
Ign:1 http://cdn-fastly.deb.debian.org/debian stretch InRelease
Atteint:3 http://security.debian.org stretch/updates InRelease
Atteint:2 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease
Atteint:4 http://cdn-fastly.deb.debian.org/debian stretch Release
Lecture des listes de paquets…
Lecture des listes de paquets…
Construction de l'arbre des dépendances…
Lecture des informations d'état…
pciutils is already the newest version (1:3.5.2-1).
procps is already the newest version (2:3.3.12-3).
python-yaml is already the newest version (3.12-1).
0 mis à jour, 0 nouvellement installés, 0 à enlever et 32 non mis à jour.
Lecture des listes de paquets…
Construction de l'arbre des dépendances…
Lecture des informations d'état…
apt-transport-https is already the newest version (1.4.8).
ca-certificates is already the newest version (20161130+nmu1).
dirmngr is already the newest version (2.1.18-8~deb9u1).
gnupg2 is already the newest version (2.1.18-8~deb9u1).
0 mis à jour, 0 nouvellement installés, 0 à enlever et 32 non mis à jour.
Warning: apt-key output should not be parsed (stdout is not a terminal)
Executing: /tmp/apt-key-gpghome.jJiab6BiSX/gpg.1.sh --keyserver-options ca-cert-file=/etc/ssl/certs/ca-certificates.crt,http-proxy=http://10.105.4.1:3128/ --fetch-keys https://repo.saltstack.com/apt/debian/9/amd64/2016.3/SALTSTACK-GPG-KEY.pub
gpg: keyserver option 'ca-cert-file' is obsolete; please use 'hkp-cacert' in dirmngr.conf
gpg: requesting key from 'https://repo.saltstack.com/apt/debian/9/amd64/2016.3/SALTSTACK-GPG-KEY.pub'
gpg: WARNING: unable to fetch URI https://repo.saltstack.com/apt/debian/9/amd64/2016.3/SALTSTACK-GPG-KEY.pub: Erreur réseau
Atteint:2 http://security.debian.org stretch/updates InRelease
Ign:1 http://cdn-fastly.deb.debian.org/debian stretch InRelease
Atteint:3 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease
Atteint:4 http://cdn-fastly.deb.debian.org/debian stretch Release
Réception de:6 https://repo.saltstack.com/apt/debian/9/amd64/2016.3 stretch InRelease [2 841 B]
Ign:6 https://repo.saltstack.com/apt/debian/9/amd64/2016.3 stretch InRelease
Réception de:7 https://repo.saltstack.com/apt/debian/9/amd64/2016.3 stretch/main amd64 Packages [4 459 B]
7 300 o réceptionnés en 0s (10,8 ko/s)
Lecture des listes de paquets…
W: Erreur de GPG : https://repo.saltstack.com/apt/debian/9/amd64/2016.3 stretch InRelease : Les signatures suivantes n'ont pas pu être vérifiées car la clé publique n'est pas disponible : NO_PUBKEY 0E08A149DE57BFBE
W: The repository 'https://repo.saltstack.com/apt/debian/9/amd64/2016.3 stretch InRelease' is not signed.
 *  INFO: Running config_salt()
 * DEBUG: The passed destination(/etc/salt) is a directory
 * DEBUG: Full destination path is now: /etc/salt/minion
 * DEBUG: Moving /tmp/.saltcloud-a2e37c35-ff2f-4bd4-a15a-d01afc19be2c/minion to /etc/salt/minion
 * DEBUG: The passed destination(/etc/salt/pki/minion/) is a directory
 * DEBUG: Full destination path is now: /etc/salt/pki/minion//minion.pem
 * DEBUG: Moving /tmp/.saltcloud-a2e37c35-ff2f-4bd4-a15a-d01afc19be2c/minion.pem to /etc/salt/pki/minion//minion.pem
 * DEBUG: The passed destination(/etc/salt/pki/minion/) is a directory
 * DEBUG onnection to 10.105.4.172 closed.
0m: Full destination path is now: /etc/salt/pki/minion//minion.pub
 * DEBUG: Moving /tmp/.saltcloud-a2e37c35-ff2f-4bd4-a15a-d01afc19be2c/minion.pub to /etc/salt/pki/minion//minion.pub
 *  INFO: Running install_debian_9_stable()
Lecture des listes de paquets…
Construction de l'arbre des dépendances…
Lecture des informations d'état…
The following additional packages will be installed:
  dctrl-tools debconf-utils javascript-common libjs-jquery libjs-sphinxdoc
  libjs-underscore libpgm-5.2-0 libsodium18 libzmq5 python-apt
  python-backports-abc python-cffi-backend python-chardet
  python-concurrent.futures python-croniter python-crypto python-cryptography
  python-dateutil python-enum34 python-idna python-ipaddress python-jinja2
  python-markupsafe python-msgpack python-openssl python-pkg-resources
  python-pyasn1 python-requests python-setuptools python-singledispatch
  python-six python-systemd python-tornado python-tz python-urllib3 python-zmq
  salt-common
Paquets suggérés :
  debtags apache2 | lighttpd | httpd python-apt-dbg python-apt-doc
  python-crypto-dbg python-crypto-doc python-cryptography-doc
  python-cryptography-vectors python-enum34-doc python-jinja2-doc
  python-openssl-doc python-openssl-dbg doc-base python-socks
  python-setuptools-doc python-mysqldb python-pycurl python-tornado-doc
  python-twisted python-ntlm python-mako salt-doc python-augeas
Les NOUVEAUX paquets suivants seront installés :
  dctrl-tools debconf-utils javascript-common libjs-jquery libjs-sphinxdoc
  libjs-underscore libpgm-5.2-0 libsodium18 libzmq5 python-apt
  python-backports-abc python-cffi-backend python-chardet
  python-concurrent.futures python-croniter python-crypto python-cryptography
  python-dateutil python-enum34 python-idna python-ipaddress python-jinja2
  python-markupsafe python-msgpack python-openssl python-pkg-resources
  python-pyasn1 python-requests python-setuptools python-singledispatch
  python-six python-systemd python-tornado python-tz python-urllib3 python-zmq
  salt-common salt-minion
0 mis à jour, 38 nouvellement installés, 0 à enlever et 32 non mis à jour.
Il est nécessaire de prendre 7 734 ko dans les archives.
Après cette opération, 36,0 Mo d'espace disque supplémentaires seront utilisés.
ATTENTION : les paquets suivants n'ont pas été authentifiés.
  python-jinja2
E: There were unauthenticated packages and -y was used without --allow-unauthenticated
 * ERROR: Failed to run install_debian_9_stable()!!!
 * DEBUG: Removing the logging pipe /tmp/bootstrap-salt.logpipe
[ERROR   ] Failed to deploy 'bootstraptest'. Error: Command 'ssh -t -t -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -oPasswordAuthentication=no -oChallengeResponseAuthentication=no -oPubkeyAuthentication=yes -oIdentitiesOnly=yes -oKbdInteractiveAuthentication=no -i /root/.ssh/id_rsa -p 22 root@10.105.4.172 '/tmp/.saltcloud-a2e37c35-ff2f-4bd4-a15a-d01afc19be2c/deploy.sh -c '"'"'/tmp/.saltcloud-a2e37c35-ff2f-4bd4-a15a-d01afc19be2c'"'"' -D -F -H http://10.105.4.1:3128/ stable 2016.3'' failed. Exit code: 1

@rallytime
Copy link
Contributor

@vutny Do you mind swinging back around here when you have a moment?

@rallytime rallytime reopened this Oct 10, 2017
@EvaSDK
Copy link
Author

EvaSDK commented Mar 21, 2018

Testing again today, both stable & develop are still broken. Also, I cannot use stable as-is now as it does not know about 2017.7 which is the current release I use on my infra.

@EvaSDK
Copy link
Author

EvaSDK commented Mar 21, 2018

Re-reading the PR, using -l is not acceptable as I don't want to get rid of SSL validations and making proxy transparent is not possible either as it comes with its own problems and this is not required by 100% of the rest of the infrastructure I manage. Can't we just go back to the old method as dirmngr is unlikely to be unbroken in the near future ?

EvaSDK added a commit to EvaSDK/salt-bootstrap that referenced this issue Apr 4, 2018
@EvaSDK
Workaround gnupg2 not being proxy friendly, refs saltstack#1137
f1f88e5 
GnuPG2 does not support proxies using the CONNECT method which means only
transparent proxies are supported for encrypted traffic or one must degrade
security by disabling encryption during provisioning.
@EvaSDK EvaSDK mentioned this issue Apr 4, 2018
Closed
@rallytime rallytime removed P2 labels Jul 11, 2018
@stale
Copy link

stale bot commented Jan 20, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

@stale stale bot added the stale label Jan 20, 2019
@stale stale bot closed this as completed Feb 7, 2019
@nielssorensen nielssorensen mentioned this issue Jun 20, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug stale
Projects
None yet
Milestone
Approved
Development

Successfully merging a pull request may close this issue.

Fix #1137: import GPG key through an HTTP(S) proxy
3 participants
@EvaSDK @rallytime @vutny