The Python Software Foundation Packaging Working Group has received a grant from Facebook Research to implement advanced security features for PyPI.
PyPI is a foundational component of the Python ecosystem and broader computer software and technology landscape. This project aims to improve the security of PyPI for all users worldwide, whether they are direct users like project maintainers and pip installers or indirect users. The impact of this work will be to implement long-desired security features for the service (see below).
We plan to begin the project in Quarter 4 of 2019. Because of the size of the project, funding has been allocated to secure one or more contractors to complete the development, testing, verification, and assist in the rollout of necessary features.
Please read this RFP and respond to let us know if you have questions, or submit a proposal if you are interested in performing the work.
| Date | Milestone |
|---|---|
| August 28 | Request for Information period opened. |
| September 18 | Request for Information period closed. |
| September 25 | Request for Proposal period opened. |
| October 21 | Request for Proposal period closes. |
| October 29 | Date proposals will have received a decision. |
| November 30 | Contracts for accepted proposals should be finalized. |
| December 2 | Contract work commences. |
A Request for Proposals (RFP) is a process intended to allow us (the Python Software Foundation) to collect proposals from potential contractors and select contractor(s) best suited to fulfill the specified work.
After the RFP period closes we will evaluate the received proposals based on the evaluation criteria, seek clarification from proposers as necessary, and select one or more contractors to complete the work specified in the scope section.
Note: This Request For Proposal document may be updated to reflect things that we learn during the process. The canonical version and history is available here.
Proposals should be submitted as Portable Document Format (PDF) files via email to ewdurbin@pyfound.org.
Proposals must be submitted before the end of the day October 21, 2019 AoE (2019-10-21T12:00:00Z).
A submission must, at a minimum, include the following elements:
-
Description of the team that will perform the work.
- General overview and names of individuals.
- Experience with relevant technologies.
- Freelance or firm? Incorporation? Subcontracting?
- Free/Open Source Software experience?
-
Agreement to project management and reporting requirements.
-
What Milestone(s) are you proposing for?
- We recommend proposing for all of the work in scope of Milestone 1, Milestone 2, or both.
-
Examples of similarly-complex projects completed previously.
- Referencing contributions to Free/Open Source projects is encouraged.
-
Project timeline estimates by milestone and task. These timelines should fit within our project timeline, however proposers may suggest adjustments based on their availability.
- Milestone 1 - Verifiable cryptographic signing of artifacts
- Milestone 2 - Systems for Automated Detection of Malicious Uploads
-
Project budget by milestone and task. Deviations from our estimated budgets and caps should be described and supported.
- Milestone 1 - Verifiable cryptographic signing of artifacts
- Milestone 2 - Systems for Automated Detection of Malicious Uploads
In total, we expect your proposal to be 5-10 pages long, but feel free to go under 5 or over 10 pages if you feel it is appropriate.
- Contains all elements specified in the elements of the proposal.
- Proposal is detailed enough to properly assess further criteria.
- Formatting and Submission requirements:
- Portable Document Format (PDF)
- Emailed to ewdurbin@pyfound.org by October 16, 2019 AoE (2019-10-16T12:00:00Z)
- Does the proposal demonstrate relevant experience necessary to complete the work?
- Is there demonstrable experience with enough of the relevant technologies for each Milestone to support timelines?
- Do the examples of similarly complex projects and any references to past Free/Open Source Software contributions indicate competency?
- Does the budget proposed fit within our estimated budgets and caps?
- Are the milestone and task budgets reasonable and competitive?
- Are any deviations from the estimated budgets and caps well supported and explained?
- Do the proposers agree to the project management and reporting requirements?
- Are the project timeline estimates within our project timeline? Are deviations from our timeline explained and supported?
This Request for Proposals is seeking backend developers to implement, test, verify, and assist in the rollout of the following features to the codebase that powers PyPI.
Discussions leading to these milestones can be read in the forum for our Request for Information.
- Implementation of PEP 458 once accepted to add integration of The Update Framework to PyPI
- Development of either a stand alone service or code in the Warehouse codebase to create, sign, serve, and handle caching concerns for TUF metadata
- Development of necessary code in the Warehouse codebase to integrate TUF metadata and signing
Relevant background and context
- Development of systems for generation and signing of metadata compliant with TUF
- Development of systems for serving signed metadata compliant with TUF
- Development of systems for appropriately caching, and invalidating caches, for metadata service
- Documentation for Administrators of PyPI for handling key material and interacting with the systems
- Documentation for PyPI client libraries for sourcing and validating TUF metadata
Note: Implementation of support for TUF signing or verification in the Python packaging toolchain (e.g. setuptools, pip, twine, etc) is not in scope.
- Development of infrastructure as either a standalone service or code in the Warehouse codebase to automatically screen metadata and uploads to PyPI for malicious content via pluggable checks, reporting results to PyPI
- Development of necessary code in the Warehouse codebase to store results of automated screening for administrator review
Relevant background and context
- Develop systems for running automated checks on new Projects, Releases, and Release Files.
- Develop a pluggable system for development and deployment of new checks
- Implement database models to store check results relating to Projects, Releases, and Release Files
- Implement Administrator views for reviewing results
- Documentation of the process for the development and submission of new checks
Note: Surfacing malware detection results to PyPI publishers and users via the Web User Interface or any API is not in scope.
Budgets and caps are provided to help contractors in preparing their proposals.
Budgets are PSF's estimates, and caps are based on the funding commitment that we have received.
Caps are provided to indicate that we understand that some features may require more funds than our estimated budget. Proposals may go over budget up to the cap for one feature, but fall under budget for another. We understand that there is the potential for significant deviation from our budgets; as long as proposals meet all criteria and fulfill requirements without going over the total budget cap that is not a problem.
Estimates and caps in United States dollars.
| Task | TBD Budget | TBD Cap |
|---|---|---|
| Verifiable cryptographic signing of artifacts (approximately 4-5 weeks) | $25,000 | $30,000 |
| Automated detection of malicious uploads (approximately 3-4 weeks) | $20,000 | $24,500 |
| Documentation for above features | $3,000 | $5,500 |
| Total | not applicable | $65,000 |
This project is intended to be completed over a three to five month period beginning December 2019. Proposals with a shorter or longer timeline are acceptable if it is explained, and supported by estimated budget and costs.
The codebase behind PyPI is called Warehouse and is licensed under the Apache License 2.0. All work submitted or dependencies added must be compliant with this license.
The backend codebase is in Python with a CSS, HTML, and JavaScript frontend (using the Stimulus framework).
Potential proposers should be comfortable with Python and may need to implement some features or views for the frontend, but will have support from an additional contractor focused on user interface and user experience design to implement CSS and HTML changes. JavaScript features may be required but resources are available to assist with this as well.
Familiarity and expertise with all technologies is not required. Strong Python skills and experience are a must, though.
For the best primer, see the developer documentation for Warehouse.
You can also see the complete codebase on GitHub.
- Python 3.6+
- Pyramid
- PostgreSQL - SQLAlchemy - psycopg2
- Redis
Note: Our frontend is primarily static; these tools power the toolchain that creates our final assets.
This project will be led and managed by the Python Software Foundation Director of Infrastructure and potentially an external project manager.
Regular meetings will be held to coordinate efforts among the project managers, backend developers, frontend developers, and UX designer.
Oral or textual status reporting during these meetings, as well as regular textual summaries of current status, will be required. Additionally, participation on the public issue tracker and submission of changes via code review for the project will be required.
Please contact Ernest W. Durbin III <ewdurbin@pyfound.org>, Director of Infrastructure at the Python Software Foundation.