Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unreachable pyup.io leads to failing `pipenv check` #2825

Closed
l0b0 opened this issue Sep 5, 2018 · 7 comments
Closed

Unreachable pyup.io leads to failing `pipenv check` #2825

l0b0 opened this issue Sep 5, 2018 · 7 comments

Comments

@l0b0
Copy link

@l0b0 l0b0 commented Sep 5, 2018

I'm working on a project which has been running a lot of pipenv checks (possibly on the order of hundreds of requests per day). I think we've been throttled at the network level, because this command now results in messages like this:

Connection to pyup.io timed out. (connect timeout=5)

curl pyup.io can't reach the site, but torify curl pyup.io (that is, via a proxy) works, and dig pyup.io returns the same IP as SSL Labs detects, so it's definitely restricted to somewhere between this network and that site.

I've changed our process to run pipenv check much less often, but would it be possible to detect or avoid this issue in pipenv? Ideas:

  • Configuring our own API key so we can reliably stay below any limits.
  • Suggesting solutions when the connection times out.
  • Configurable caching of the security issues database. I realize this is probably a very risky move since security issues may have to be fixed within hours, but it could be useful for people who for process reasons end up running a lot of checks.

PS: The recommended forum for these requests returns HTTP 404.

@uranusjr
Copy link
Member

@uranusjr uranusjr commented Sep 5, 2018

Allowing user configuration may be the simplest way to go. You can set it to something else (default to our built-in key if not set), or (maybe) a special value to disable pyup checks completely. At least the first part would be fairly straightforward, and contributions would be welcomed :)

@techalchemy
Copy link
Member

@techalchemy techalchemy commented Nov 4, 2018

Since we have merged #2835 I believe we can close this one out, thanks for the report / hope the fix works for your use case!

@techalchemy techalchemy closed this Nov 4, 2018
@leotrubach
Copy link

@leotrubach leotrubach commented Feb 16, 2019

Currently pyup.io gives 500 error and we need to distinguish the following situations:

  1. site is unreachable (okay, we can skip in this case)
  2. there are outdated packages (we need to update packages)

This could be done by returning different error codes
@techalchemy could you please reopen this issue?

@leotrubach
Copy link

@leotrubach leotrubach commented Feb 16, 2019

As a workaround this one could be used:

pipenv check || if [ $(curl --write-out %{http_code} --silent --output /dev/null pyup.io) -eq 500 ]; then echo "Site unreachable. Skipping check"; else false; fi
@leotrubach
Copy link

@leotrubach leotrubach commented Feb 16, 2019

Hmm... I couldn't really figure out what's wrong. But when running pipenv check it fails but safety check succeeds

@leotrubach
Copy link

@leotrubach leotrubach commented Feb 16, 2019

Strace output

Checking installed package safety…
read(9, "", 50000)                      = 0
read(5, "Unable to load vulnerability database\n", 32768) = 38
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2189, si_uid=1001, si_status=255, si_utime=0, si_stime=0} ---
read(5, "", 32768)                      = 0
read(7, "", 32768)                      = 0
An error occurred:

@mcallaghan-bsm
Copy link

@mcallaghan-bsm mcallaghan-bsm commented Jul 19, 2019

As a workaround this one could be used:

pipenv check || if [ $(curl --write-out %{http_code} --silent --output /dev/null pyup.io) -eq 500 ]; then echo "Site unreachable. Skipping check"; else false; fi

Is this really a good workaround?
You don't want your pipeline security checks "false passing".

(of note, our workaround is that we have our CI jobs "allow fail = true" which converts the gitlab job into a WARNING instead, therefore not blocking the entirety of the pipeline -- this way the TRUTH is reported still, but does not block development during outages)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants