Mention RSS feed in FAQ, security documentation

[WhyNotHugo]

Over the last couple of years, we've seen a few instances of malicious packages, (eg: pypi/legacy#644, http://www.nbu.gov.sk/skcsirt-sa-20170909-pypi/).

Some approaches out there validate package signatures and alike, but don't really address the root cause, that just makes sure nobody's tampering valid packages, but typosquatting and similar attacks are still possible.

#2268 addresses a very common form of attack, but other similar things might still show up.

I'd like to propose something similar to what CAs do with Certificate Transparency; have PyPI/Warehouse publish a list of all newly registered packages (I don't think updates of existing ones are relevant for this).
If it's in some format where we can check that the log hasn't been tampered, it's a nice addition, but not strictly required for the current intention.

This would allow both devs and third parties to keep an eye (my just looking at the list periodically) or perform other forms of automated checks for potential malicious packages.

[di]

There's currently an RSS feed which publishes all newly created packages, with their name, title, description and a link to the project on PyPI:

https://pypi.org/rss/packages.xml

To my knowledge, nobody is using this to vet the names of new packages, but I don't see why it couldn't serve this purpose for third parties.

[dstufft]

PyPI actually published a "journal" of everything that happens on PyPI, the older API is since a specific timestamp, but the recommend API is since a specific changelog ID.

It's an XMLRPC api (sadly) but you can basically call changelog_last_serial() to get the current revision of the journal, then look at /simple/ to get a list of all packages that currently exist, then from there on out you can call changelog_since_serial(since_serial) with the current serial to get the list of all actions that have occurred since your last serial.

Another alternative is to just do a GET /simple/, and record the ETag, and then periodically do a conditional HTTP GET to /simple/ with that Etag included. If you get a 200 then something was added or removed and if you get a 304then nothing has changed. I'm not 100% sure that currently works completely, but I think it does.

Another alternative is the RSS feed as @di mentioned.

There is also a planned feature of web hooks that can trigger whenever something happens, you can see more about that at #360.

Do either of those satisfy this use case?

[WhyNotHugo]

I think a combination of RSS and #360 would suffice for this scenario. I quickly looked through the docs and didn't find a reference to this.

I think this feature request can be really summed down to: let's mention the above (and, potentially, #360, once done), in the security part of the doc or website, for better discoverability (and to actually promote these being monitored by someone).

[brainwane]

Thanks for the suggestion, @WhyNotHugo! I have retitled the issue appropriately.

So, someone would update warehouse/templates/pages/help.html and docs/security.rst to mention the RSS feeds, which now have their own documentation at docs/api-reference/, and to mention the tips in #2413 (comment) .

Good First Issue: This issue is good for first time contributors. If there is not a corresponding pull request for this issue, it is up for grabs. For directions for getting set up, see our Getting Started Guide. If you are working on this issue and have questions, please feel free to ask them here, in #pypa-dev on Freenode, or on the pypa-dev mailing list.

[brainwane]

Now that #3173 has been merged, https://warehouse.readthedocs.io/security/ and https://test.pypi.org/help/#APIs point to our API documentation, which now includes details about accessing PyPI's journal. I therefore think we can close this issue; thank you for opening it, @WhyNotHugo!

[floer32]

Great! 🎆

[WhyNotHugo]

Thanks!