Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Saltstack repository - status page/public mirror possibility? And causing stucked install states? #46021

Open
Reiner030 opened this issue Feb 13, 2018 · 22 comments
Labels
Feature new functionality including changes to functionality and code refactors, etc.
Milestone

Comments

@Reiner030
Copy link

Hello,

It would be great to have

  • a public status page to easily recognize down/routing/ddos/... issues
  • the possibilty for public mirrors like most OSS offers so it's easy
    • to lower the impact of data transfers for you / make traffic more "localized"
    • switch to (other) mirrors if such failure occurs

Actual (over 3-4 hours yet) the saltstack repository repo.saltstack.com is mainly not available in Germany - different locations tested for the case it's geoip distributed... so

  • fresh setup is not possible
  • but also other states which tries to setup packages get stucked for hours... (I killed 1st run av e.g.:
[INFO    ] Executing state pkg.installed for [python-redis]
[DEBUG   ] Could not LazyLoad pkg.normalize_name: 'pkg.normalize_name' is not available.
[DEBUG   ] Could not LazyLoad pkg.check_db: 'pkg.check_db' is not available.
[DEBUG   ] Could not LazyLoad pkg.normalize_name: 'pkg.normalize_name' is not available.
[INFO    ] Executing command ['dpkg', '--get-selections', '*'] in directory '/root'
[INFO    ] Executing command ['systemd-run', '--scope', 'apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'python-redis'] in dire[...]

which is very weird...

I was checking the cause of this stucked states (running salt-call 2017.7.2 (Nitrogen)) which keeps apt locked (latest Debian Stretch Cloud image).

As state the installs stopped working; it seems after the package install routine itself but apt/history log has no finished entry written.
And apt-get is so busy that even a kill -9 $(pidof apt-get) is not cancelling the call which is extremely unknown behavior for such tasks ... => Perhaps it comes because of running it as child of systemd-run ?
When running the install manually which I tested for comparison / check if there is an apt problem it's all fine there.

@Reiner030
Copy link
Author

Now it seems up again.

BTW: There is a mirror offer but the rsync would include all distributions (OS) and versions included which may results in some hundreds GBs / some TBs of (mostly unwanted) files which is not very useful for "local mirror" setups - only for public ones (can you setup some info files about the sizes, too?):
https://repo.saltstack.com/#mirror

For Debian/Ubuntu I know about apt-cacher-ng which would fit much better.
For Redhad based repositories it seems best to use the combination reposync/createrepo (short-checked - not using this distro's)

@evarghese
Copy link

It seems we are running into an ssl error:

curl https://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/dists/trusty/main/binary-i386/Packages
curl: (35) Unknown SSL protocol error in connection to repo.saltstack.com:443

This is only happening intermittently, but it is hanging apt as Reiner030 reported.

@deepakhj
Copy link

deepakhj commented Feb 14, 2018

I'm having issues with server certificate verification. Some time today, I am unable to bootstrap new hosts.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   264  100   264    0     0   3435      0 --:--:-- --:--:-- --:--:--  3473
100  244k  100  244k    0     0  1314k      0 --:--:-- --:--:-- --:--:-- 1314k
root@ip-xxx-xx-xx-xx:~# sudo sh install_salt.sh -P
sudo: unable to resolve host ip-xxx-xx-xx-xx
 *  INFO: Running version: 2017.12.13
 *  INFO: Executed by: shell pipe
 *  INFO: Command line: 'install_salt.sh -P'

 *  INFO: System Information:
 *  INFO:   CPU:          GenuineIntel
 *  INFO:   CPU Arch:     x86_64
 *  INFO:   OS Name:      Linux
 *  INFO:   OS Version:   3.13.0-119-generic
 *  INFO:   Distribution: Ubuntu 14.04

 *  INFO: Installing minion
 *  INFO: Found function install_ubuntu_stable_deps
 *  INFO: Found function config_salt
 *  INFO: Found function preseed_master
 *  INFO: Found function install_ubuntu_stable
 *  INFO: Found function install_ubuntu_stable_post
 *  INFO: Found function install_ubuntu_restart_daemons
 *  INFO: Found function daemons_running
 *  INFO: Found function install_ubuntu_check_services
 *  INFO: Running install_ubuntu_stable_deps()
Ign http://us-east-1.ec2.archive.ubuntu.com trusty InRelease
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates InRelease
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports InRelease
Hit http://us-east-1.ec2.archive.ubuntu.com trusty Release.gpg
Hit http://us-east-1.ec2.archive.ubuntu.com trusty Release
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/main Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/restricted Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/universe Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/multiverse Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/main amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/restricted amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/universe amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/multiverse amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/multiverse Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/restricted Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/universe Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/main Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/restricted Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/universe Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/multiverse Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/main amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/restricted amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/universe amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/multiverse amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/main Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/multiverse Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/restricted Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/universe Translation-en
Hit http://security.ubuntu.com trusty-security InRelease
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/main Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/restricted Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/universe Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/multiverse Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/main amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/restricted amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/multiverse amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/main Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/multiverse Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/restricted Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/universe Translation-en
Hit http://security.ubuntu.com trusty-security/main Sources
Ign http://us-east-1.ec2.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us-east-1.ec2.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us-east-1.ec2.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us-east-1.ec2.archive.ubuntu.com trusty/universe Translation-en_US
Hit http://security.ubuntu.com trusty-security/universe Sources
Hit http://security.ubuntu.com trusty-security/main amd64 Packages
Hit http://security.ubuntu.com trusty-security/universe amd64 Packages
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Ign https://repo.saltstack.com trusty InRelease
Ign https://repo.saltstack.com trusty Release.gpg
Ign https://repo.saltstack.com trusty Release
Err https://repo.saltstack.com trusty/main amd64 Packages
  server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
Ign https://repo.saltstack.com trusty/main Translation-en_US
Ign https://repo.saltstack.com trusty/main Translation-en
W: Failed to fetch https://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/dists/trusty/main/binary-amd64/Packages  server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

E: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
apt-transport-https is already the newest version.
ca-certificates is already the newest version.
gnupg-curl is already the newest version.
The following package was automatically installed and is no longer required:
  grub-pc-bin
Use 'apt-get autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 104 not upgraded.
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.EEw1Qz5MYk --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver-options ca-cert-file=/etc/ssl/certs/ca-certificates.crt --fetch-keys https://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/SALTSTACK-GPG-KEY.pub
gpgkeys: https fetch error 60: server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
gpg: keyserver internal error
gpg: WARNING: unable to fetch URI https://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/SALTSTACK-GPG-KEY.pub: keyserver error
 * ERROR: Failed to run install_ubuntu_stable_deps()!!!```

@lsh-0
Copy link

lsh-0 commented Feb 14, 2018

similar error here:

# apt-get -q update
[...]
Ign https://repo.saltstack.com trusty/main amd64 Packages/DiffIndex
Ign https://repo.saltstack.com trusty/main Translation-en
Err https://repo.saltstack.com trusty/main amd64 Packages
  server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
Fetched 1198 kB in 3min 36s (5544 B/s)
[ERROR   ] stderr: W: Failed to fetch https://repo.saltstack.com/apt/ubuntu/14.04/amd64/archive/2016.3.6/dists/trusty/main/binary-amd64/Packages  server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

E: Some index files failed to download. They have been ignored, or old ones used instead.
[ERROR   ] retcode: 100
[ERROR   ] An error was encountered while installing package(s): W: Failed to fetch https://repo.saltstack.com/apt/ubuntu/14.04/amd64/archive/2016.3.6/dists/trusty/main/binary-amd64/Packages  server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

E: Some index files failed to download. They have been ignored, or old ones used instead.

@lsh-0
Copy link

lsh-0 commented Feb 14, 2018

I can replicate @evarghese case:

# curl -vvv https://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/dists/trusty/main/binary-i386/Packages
* Hostname was NOT found in DNS cache
*   Trying 138.197.226.47...
*   Trying 2604:a880:400:d0::2:e001...
* Immediate connect fail for 2604:a880:400:d0::2:e001: Network is unreachable
* Connected to repo.saltstack.com (138.197.226.47) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
* 	 subject: C=US; ST=UT; L=Lehi; O=Salt Stack, Inc.; CN=*.saltstack.com
* 	 start date: 2017-05-08 00:00:00 GMT
* 	 expire date: 2019-05-13 12:00:00 GMT
* 	 subjectAltName: repo.saltstack.com matched
* 	 issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
* 	 SSL certificate verify ok.
> GET /apt/ubuntu/14.04/amd64/latest/dists/trusty/main/binary-i386/Packages HTTP/1.1
> User-Agent: curl/7.35.0
> Host: repo.saltstack.com
> Accept: */*
> 
* SSL read: error:00000000:lib(0):func(0):reason(0), errno 104
* Closing connection 0
curl: (56) SSL read: error:00000000:lib(0):func(0):reason(0), errno 104

update: curl case is now working, apt-get isn't.

update2: apt-get is now working too

+1 for a status page

@garethgreenaway garethgreenaway added this to the Blocked milestone Feb 14, 2018
@garethgreenaway garethgreenaway added the Pending-Discussion The issue or pull request needs more discussion before it can be closed or merged label Feb 14, 2018
@garethgreenaway
Copy link
Contributor

@dubb-b FYI.

@xavieryao
Copy link

Would it be possible to maintain a "official mirror site" or "alternative download" list? We are hosting one at https://mirrors.tuna.tsinghua.edu.cn/saltstack/ and wish to be displayed one the official webpage.

@stale
Copy link

stale bot commented Jan 9, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

@stale stale bot added the stale label Jan 9, 2020
@Reiner030
Copy link
Author

yes, we'll still want to have a status page created from you showing your service status...

@stale
Copy link

stale bot commented Jan 9, 2020

Thank you for updating this issue. It is no longer marked as stale.

@stale stale bot removed the stale label Jan 9, 2020
@stale
Copy link

stale bot commented Feb 8, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

@stale stale bot added the stale label Feb 8, 2020
@Reiner030
Copy link
Author

the stable bot... the new variant of "bugfixing by ignoring"...

@stale
Copy link

stale bot commented Feb 8, 2020

Thank you for updating this issue. It is no longer marked as stale.

@stale stale bot removed the stale label Feb 8, 2020
@stale
Copy link

stale bot commented Mar 9, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

@stale stale bot added the stale label Mar 9, 2020
@Reiner030
Copy link
Author

And the bot is still very obstinate in closing wanted features... we should consider implement a ping-ping game on it...

@stale
Copy link

stale bot commented Mar 9, 2020

Thank you for updating this issue. It is no longer marked as stale.

@stale stale bot removed the stale label Mar 9, 2020
@stale
Copy link

stale bot commented Apr 8, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

@stale stale bot added the stale label Apr 8, 2020
@Reiner030
Copy link
Author

ping

@stale
Copy link

stale bot commented Apr 8, 2020

Thank you for updating this issue. It is no longer marked as stale.

@stale stale bot removed the stale label Apr 8, 2020
@garethgreenaway
Copy link
Contributor

@Reiner030 Apologies for the delay on this one. @bryceml @felippeb Any thoughts on providing some sort of status page for the repo site? Looping @saltstack/team-core in on this too.

@sagetherage sagetherage removed this from the Blocked milestone Aug 13, 2020
@sagetherage sagetherage added needs-triage and removed Pending-Discussion The issue or pull request needs more discussion before it can be closed or merged labels Aug 13, 2020
@sagetherage sagetherage added Feature new functionality including changes to functionality and code refactors, etc. and removed needs-triage labels Aug 13, 2020
@sagetherage sagetherage added this to the Approved milestone Aug 13, 2020
@bryceml
Copy link
Contributor

bryceml commented Aug 13, 2020

This should be less of an issue now that we have cloudfront in front of s3. It could still be useful though. We'll discuss it.

@whytewolf
Copy link
Contributor

we should just point at the aws status page. cause when aws down salt repo is down.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature new functionality including changes to functionality and code refactors, etc.
Projects
None yet
Development

No branches or pull requests

10 participants