Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Roadmap update for TUF support #5247

Open
LucidOne opened this issue Jan 4, 2019 · 24 comments
Open

Roadmap update for TUF support #5247

LucidOne opened this issue Jan 4, 2019 · 24 comments

Comments

@LucidOne
Copy link

@LucidOne LucidOne commented Jan 4, 2019

Is it possible to get an update on the development roadmap about when TUF or other encryption support might be deployed? Thanks!

Also, it appears that tomorrow will be 6 years since January 5, 2013.

@nealmcb

This comment has been minimized.

Copy link

@nealmcb nealmcb commented Feb 19, 2019

Thanks - good question.
I note that TUF is one of the options noted under Add support for API keys · Issue #994 · pypa/warehouse which appears under the "Cool but not urgent" milestone.
I agree that TUF addresses many of the nicely described PyPI-specific concerns that @dstufft wrote way back on 23 July 2013: Why Package Signing is not the Holy Grail

@LucidOne

This comment has been minimized.

Copy link
Author

@LucidOne LucidOne commented Feb 19, 2019

I believe that the package signing issues stated can be resolved and in 2019 our internet depends on the security of infrastructure such as PyPI. If there is not a roadmap for TUF support, I'm going to look at solving the problem.

@MyNameIsCosmo

This comment has been minimized.

Copy link

@MyNameIsCosmo MyNameIsCosmo commented Feb 19, 2019

Perhaps we can evaluate integration with third-party services for verification?
Keybase.io helps create trust by building a verified profile attached to a GPG key across multiple services.
GitHub handles gpg signature verification for commits, refs, releases, etc. (although, your GH pgp key is kept in your GH settings... 2fa could help mitigate unauthorized account access)
Falling back, you can have a PGP key on MIT's PGP keybase which forwards a public key to key servers around the world.

Now, this does assume putting trust in third-party services themselves (and the author of the actual PGP key), but the chances are very slim that your PGP key would be replaced on multiple services at once. The biggest issue is an author being unable to secure their private key (no password, weak password, bad private key handling, etc...), or their account becoming compromised (e.g. GitHub).

This doesn't solve the problem on PyPi's side of verifying someone, although PyPi shouldn't have to verify anyone. PyPi should host the package. It should display information about the package being signed (or unsigned), the package's origins and contents, and whoever is downloading the package should check that key against their trusted sources.
In the past, PyPi has hosted malicious code. Of course, this happens. It is expected. Other package managers suffer as well.


I'm sure we all can go on about this.
Sure, PGP signing doesn't guarantee security. It doesn't guarantee verification. It doesn't even guarantee safe, robust, reliable, ... code. PGP is just a part of a solution to of a complex problem.
As signing becomes supported by major services (package maintainers, source code repositories, document services, ...), a trust network (keybase, key servers, ...) can be utilized to help security-minded users track where their packages are coming from.

If a user or a maintainer doesn't care about signing, it doesn't have to be enforced at the beginning, If the feature is there, people will use it.

Edit:
A great talk - How Much Do You Trust That Package?

@LucidOne

This comment has been minimized.

Copy link
Author

@LucidOne LucidOne commented Feb 19, 2019

I am convinced that this problem is tractable and I think the key here is that EigenTrust can provide us a mathematical model to get started.

We already have some data in the git commit history. When someone signs a git commit there is an eigentrust metric that can be calculated for all of the previous signed commits. There is reason to believe that causality and techniques like Bayesian inference may also be useful here where multiple commits are signed by a key we do not yet trust, but are temporally followed by a key we do trust. We can also federate our memories of git history and checksums to detect anomalies and attacks on infrastructure.

Sites like github and sr.ht can already provide trust metrics by validating that an email address is connected to a GPG key. We can formalize and automate our existing manual heuristics for validating packages.

Further I think we should start thinking about metadata for developers and organizations that produce software. Organizations can maintain a META-DEV repository that contains the PGP signing keys of active developers, key revocation information, and per repository release manager designation. Developers can also maintain a META-DEV repository that contains yaml (or whatever) metadata linking to a developers twitter / blog / instagram / keybase / identi.ca / matrix / Mastodon / xmpp / whatever where the PGP fingerprint can be posted.

Perhaps we need to start building keyrings in OS native packaging formats (.deb, .rpm, etc) so that trust can at least be established for the most critical python packages.

This is a complex problem but we need to take what small steps forward that we can instead of waiting another year to secure PyPI or even bother to figure out certificate pinning.

@brainwane

This comment has been minimized.

Copy link
Member

@brainwane brainwane commented Mar 22, 2019

Thank you to everyone who's raised this issue and shared their thoughts and useful resources! And sorry for the slow response!

Short answer: we'll be discussing TUF & Warehouse much more in April.

Longer answer:

The folks working on Warehouse have gotten funding to concentrate on improving Warehouse's security, and have kicked off work (funded by the Open Technology Fund) towards multi-factor auth, API keys, and an audit trail. And -- to quote the blog post --

Facebook... has provided the Python Software Foundation with a monetary gift that will be used to fund the development and deployment of enhanced security features to PyPI....

The PSF Packaging Working Group plans to use these funds to implement highly requested security features in PyPI such as cryptographic signing and verification of files uploaded and installed from the index. Additionally, systems for the automated detection of malicious uploads will lower the time to response and improve the resiliency of PyPI against attacks such as "pytosquatting".

This work will be undertaken in the second half of 2019 but planning will begin in the second quarter of the year.

We anticipate that in mid-April (so, basically within about a month) we'll be announcing a formal Request For Information to ask people to tell us about their interest in being contracted to do this work, and that part of that discussion will be further, more detailed conversations about whether TUF is the right tool for this job. So please watch for that, on this issue and on https://discuss.python.org/c/packaging .

(cc @pradyunsg since I think you're interested in this.)

@JustinCappos

This comment has been minimized.

Copy link

@JustinCappos JustinCappos commented Mar 28, 2019

From the TUF side, we're very interested in moving this forward. Let us know what we can do to help!

@trishankatdatadog

This comment has been minimized.

Copy link
Member

@trishankatdatadog trishankatdatadog commented Apr 3, 2019

Same, happy to help with this, just let us know how.

@westurner

This comment has been minimized.

Copy link

@westurner westurner commented Apr 18, 2019

"PyPI security work: multifactor auth progress & help needed"
https://discuss.python.org/t/pypi-security-work-multifactor-auth-progress-help-needed/1042/10

@brainwane

This comment has been minimized.

Copy link
Member

@brainwane brainwane commented May 23, 2019

At PyCon sprints several people spoke about the potential future of TUF in Warehouse and Python packaging, and put notes at https://docs.google.com/document/d/1Wz2-ECkicJgAmQDxMFivWmU2ZunKvPZ2UfQ59zDGj7g/ .

@trishankatdatadog

This comment has been minimized.

Copy link
Member

@trishankatdatadog trishankatdatadog commented May 23, 2019

@ofek

This comment has been minimized.

Copy link
Contributor

@ofek ofek commented May 23, 2019

I'll also devote whatever time is necessary to get this done

@lukpueh

This comment has been minimized.

Copy link
Contributor

@lukpueh lukpueh commented May 24, 2019

Thanks for putting together our notes, @brainwane! It was a pleasure meeting you guys at PyCon. @ewdurbin, any news on the RFI?

@ofek and @trishankatdatadog, your help will be very much appreciated.

@brainwane

This comment has been minimized.

Copy link
Member

@brainwane brainwane commented Aug 28, 2019

Please check out the newly posted Request for Interest regarding upcoming work implementing cryptographic signing and malware detection on PyPI.

Our current timeline:

Date Milestone
August 28 Request for Information period opens.
September 18 Request for Information period closes.
September 23 Request for Proposal period opens.
October 16 Request for Proposal period closes.
October 29 Date proposals will have received a decision.
November 30 Contracts for accepted proposals should be finalized.
December 2 Contract work commences.

And then we intend to complete the project over a three to five month period, beginning December 2019.

We're hoping to get participation from potential participants and other experts in the discussion forum, especially about implementation questions, including which of the TUF PEPs (if either) to implement!

@brainwane

This comment has been minimized.

Copy link
Member

@brainwane brainwane commented Sep 26, 2019

See the PSF's new blog post & the open RFP. Later this year, PyPI wants to start:

  • Implementation of PEP 458 once accepted to add integration of The Update Framework to PyPI
  • Development of either a stand alone service or code in the Warehouse codebase to create, sign, serve, and handle caching concerns for TUF metadata
  • Development of necessary code in the Warehouse codebase to integrate TUF metadata and signing

This means we need to move PEP 458 from "Deferred" to "Accepted" status. Per @ewdurbin's guidance, this means we'll need to get PEP 458 revised, as necessary, to pin down specifics, such as key distribution (who, where, how many?) plus any technical choices that TUF leaves up to implementations. To revise PEP 458 and get it accepted, we'll need to collaborate with previous implementers and other experts.

Given the RFP timeline the latest we should get the PEP accepted is 2 December 2019, but I'd much prefer we get it accepted by mid-October.

@trishankatdatadog

This comment has been minimized.

Copy link
Member

@trishankatdatadog trishankatdatadog commented Sep 26, 2019

Thanks for the update, @brainwane!

@JustinCappos @lukpueh Ok, so we have our work ahead of us. I have work obligations to meet, but can devote whatever time I can for this. Let's plan ASAP.

@ofek

This comment has been minimized.

Copy link
Contributor

@ofek ofek commented Sep 26, 2019

Let me know if you need more assistance, I'd be glad to help!

brainwane added a commit to brainwane/peps that referenced this issue Sep 26, 2019
Facebook Research has now funded implementation of
cryptographic signing of packages on PyPI. Per
pypa/warehouse#5247 (comment)
this means that PEP 458 now moves out of Deferred
status and into Draft status.

Since the PEP was created, the BDFL-Delegate for
PyPI-related PEPs has shifted, and Donald Stufft
is now the Delegate.
brettcannon added a commit to python/peps that referenced this issue Sep 26, 2019
Facebook Research has now funded implementation of
cryptographic signing of packages on PyPI. Per
pypa/warehouse#5247 (comment)
this means that PEP 458 now moves out of Deferred
status and into Draft status.

Since the PEP was created, the BDFL-Delegate for
PyPI-related PEPs has shifted, and Donald Stufft
is now the Delegate.
@brainwane

This comment has been minimized.

Copy link
Member

@brainwane brainwane commented Sep 26, 2019

Now that the PEP is back in Draft status*, I think the next steps are for one or more of the PEP authors to:

@dstufft is now the BDFL-Delegate for this PEP so it'll have to be the other authors (@trishankatdatadog, @vladimir-v-diaz, @JustinCappos) who push this forward. If we want to get any revisions done and get Donald to accept the PEP by mid-October then you should start the steps above in the next couple days, in my opinion.

* the version at python.org needs to be re-generated, but python/peps#1177 was accepted

@brainwane

This comment has been minimized.

Copy link
Member

@brainwane brainwane commented Sep 30, 2019

A few of us had a chat today and are working to update the PEP (python/peps#1178 is part of that), and one or more of the PEP authors will be reaching out to @ewdurbin with a few questions.

@pradyunsg

This comment has been minimized.

Copy link
Member

@pradyunsg pradyunsg commented Sep 30, 2019

@brainwane FYI - I'm happy to help with implementing functionality on the client side (i.e. pip) when we get to that point.

I think we'd want to create a tracking issue on pip's issue tracker to have implementation related discussions there, after the PEP is accepted (AFAICT how clients interact with TUF-enabled PyPI is covered by the PEP and would be discussed in the discussions on discuss.python.org).

@trishankatdatadog

This comment has been minimized.

Copy link
Member

@trishankatdatadog trishankatdatadog commented Oct 1, 2019

Hi @ewdurbin and @dstufft, we have a few questions for which we could use your help:

  1. By "key distribution," do you mean who manages how many keys and where on the PyPI side? Do you also mean how package managers such as pip would determine which keys to trust in the first place?
  2. What is the current deployment process for Warehouse? This will help us determine how to edit the PEP to discuss how to update the TUF-specific code in Warehouse.
  3. What else would you like to see more about, or see changed in the PEP?

Thanks for your time!

@trishankatdatadog

This comment has been minimized.

Copy link
Member

@trishankatdatadog trishankatdatadog commented Oct 1, 2019

@ewdurbin @dstufft I have also sent you both an email about a conference call this week, if possible. Thanks!

@brainwane

This comment has been minimized.

Copy link
Member

@brainwane brainwane commented Nov 5, 2019

Current status: python/peps#1203 is awaiting review from @dstufft to revise PEP 458. After that, there needs to be a discussion on https://discuss.python.org/c/packaging to get the PEP from "Draft" to "Accepted".

In order to make implementation easier, Dustin wants to work towards implementing #726 (removing Test PyPI from our infrastructure will make key stuff far easier). @di will be speaking more on that in the relevant issue soon.

And, starting in December, @ewdurbin will be managing the contractors who will implement TUF on PyPI. Then the first big key ceremony will be in April at PyCon North America -- if you haven't put PyCon on your calendar yet, you probably should! Conference registration will open later this month.

@brainwane

This comment has been minimized.

Copy link
Member

@brainwane brainwane commented Nov 26, 2019

The TUF team is now responding to Donald's reviews, and has a discuss.python.org thread up where people can discuss the draft PEP and suggest it be accepted.

PyCon 2020 registration is now open. Tickets do sell out, so if you intend to go, you should register soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants
You can’t perform that action at this time.