Add 'pip-compile --upgrade-insecure' option?

[jayfk]

I've been working on a database for dependencies with known security vulnerabilities. Initially for pyup.io, but since a couple of months also available as an open (raw) database at safety-db with a command line tool called safety.

Safety's command line interface checks a virtualenv or requirement files for releases with security issues and shows a report what's vulnerable. The internal API could be used by pip-compile (with the --upgrade-insecure flag) to get a list of what's vulnerable and only update affected dependencies.

How would this work?

Let's say I have a requirements.in with Django and requests. My last pip-compile run is a couple of weeks old, resulting in a slightly outdated requirements.txt.

Now a new Django security hotfix is released. I want to upgrade Django as soon as possible, but I don't want to upgrade every other dependency as well.

Running pip-compile --upgrade-insecure would only update Django.

Example

For a requirements.in

Django
requests

with a (outdated) requirements.txt

django==1.10.4         # <- outdated, known security issues
pytz==2017.1           # via django <- outdated, but no known security issues
requests==2.12.0       # <- outdated, but no known security issues

this would produce a new requirements.txt like this:

django==1.11           # <- pinned to latest
pytz==2017.1           # via django <- still outdated, not changed
requests==2.12.0       # <- still outdated, not changed

---

I've checked the code and worked on a POC, but it's a bit more complicated than I expected initially. The problem is that pip-compile has no state of the file that's being updated, it simply overwrites it.

In order for this feature to work, I'd need to add a function that is able to read the file that's about to be updated and compare this against the packages with security issues.

No rocket science, but a fair amount of work. If this is considered a useful feature to have, I'd be happy to put it together.

Thoughts?

[jayfk]

One thing I forgot to add: In terms of dependencies, both tools are largely compatible. Safety pulls in requests, but we could probably make this an extra with pip-tools[safety] anyway.

[davidovich]

I am not sure that we want to add coupling to another library. I believe not every package will adhere to the safety-db, hence making this feature only applicable on a subset of packages.

Why not use the two tools in sequence and weave them by the shell ?

I don't think pip-tools should favor any external commercial service.

Others may have differing ideas, but I am -1 on this.

[jayfk]

I believe not every package will adhere to the safety-db, hence making this feature only applicable on a subset of packages.

I'm not entirely sure what you mean by that. There's nothing to adhere to, packages are added largely by hand. Could you elaborate?

[davidovich]

Because the db is updated by hand (as I understand, the package maintainer must opt-in, hence adhere), it cannot be perfect, hence giving an --upgrade-insecure option the possibility to lie for packages not part of the db.

If package A is insecure, but not part of the db, pip-compile with the option flag will do nothing on that package, but will suggest to the user that their packages are ok.

In other words, there is too much chance of giving a false security assessment. It cannot be generalized reliably.

Leaving the responsibility to the user to use safety for his curated packages is perfectly fine, but I don't think this should be baked in pip-tools as a general option.

[jayfk]

Because the db is updated by hand (as I understand, the package maintainer must opt-in, hence adhere), it cannot be perfect, hence giving an --upgrade-insecure option the possibility to lie for packages not part of the db.

Ah, that's the part I was missing. The package maintainer has nothing to do with that. It's a system that monitors every package on PyPi for changelogs and commit messages and flags everything that could be related to some kind of security issue. 97% are false positives, so every entry is reviewed by hand. That's what I was referring to.

I agree with you that this kind of databases are never perfect. There will always be false positives or packages missing. If you take the CVE db for example, there's very little on Python packages in general. There are a couple of entries for Django, the Zope universe and 1-2 on requests. That's about it. I wouldn't conclude that this gives a false security assessment, tough. It should be used as something to check against, not as some kind of insurance or an ultimate source of truth.

I've created a dataset a while ago that contains almost every requirements file (in common places) on public GitHub repos and ran some statistics on it. I don't have exact numbers in my head, but close to 50% of all requirement files had at least one dependency with a known and documented security vulnerability. This has also to do with the tools we have available. People just don't know about the security issues.

This is where the idea was born to make this an optional thing for a widely available tool I'm using myself :).

[graingert]

@jayfk is the code to generate safety-db free software? If this were available I can see this being integrated.

[rpkilby]

@graingert - From the safety docs:

By default it uses the open Python vulnerability database Safety DB, but can be upgraded to use pyup.io's Safety API using the --key option.

And from safety-db:

The data is made available by pyup.io and synced with this repository once per month.

which reads to me as... the open database is updated monthly from pyup.io, which contains the most current information (not the other way around).

Either way, I'd be hesitant to add a dependency on a service that is not part of python's core infrastructure, especially one that seems to have a bus factor of 1. 😕

[rpkilby]

It would make more sense to me that safety/pyup provides a tool for modifying requirements files generally.
Users of pip-tools would have a workflow of:
write requirements.in > modify w/ safety > pip-compile

Those who don't use pip-tools would then just do:
write requirements.txt > modify w/ safety

[graingert]

@jayfk if you made the code that generates safe-db available under a free software licence I can see this as something that would be very easy to add

[jayfk]

Either way, I'd be hesitant to add a dependency on a service that is not part of python's core infrastructure, especially one that seems to have a bus factor of 1.

Help out :). I have a pending PR that adds a feature to warehouse (the new PyPi) allowing to deprecate a package and/or flag is as insecure: pypi/warehouse#1462 That's something Safety could use in the long run.

I'd be happy to work with more people on securing dependencies and to make packaging a safer place in general. It has been a pretty lonely road so far, but there's plenty room available.

[davidovich]

Closing as not general.