From 547ef1a05cb4ad78e9e6ebbbbf440402bf595d14 Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Thu, 8 Mar 2018 10:28:15 -0500 Subject: [PATCH] Add journal usage guidance, remove download methods in API docs (#3173) * Add security and feed information to docs Ref. #1323 * Update XML-RPC API docs fields, usage info After testing, update fields and briefly describe responses. * Update API docs since PyPI no longer offers download stats * Add API link to user help Ref: #2413. --- docs/api-reference/feeds.rst | 52 ++++++++++++++++++++++- docs/api-reference/legacy.rst | 1 + docs/api-reference/xml-rpc.rst | 65 ++++++++++++++++------------- docs/security.rst | 7 ++++ warehouse/templates/pages/help.html | 8 ++++ 5 files changed, 103 insertions(+), 30 deletions(-) diff --git a/docs/api-reference/feeds.rst b/docs/api-reference/feeds.rst index d325640b243f..af07cf547395 100644 --- a/docs/api-reference/feeds.rst +++ b/docs/api-reference/feeds.rst @@ -2,7 +2,7 @@ Feeds ===== PyPI offers two RSS feeds, the `Newest Packages Feed`_ and the `Latest Updates -Feed`_. +Feed`_. You can also call its APIs to get more details on project activity. Newest Packages Feed @@ -19,3 +19,53 @@ Latest Updates Feed Available at https://pypi.org/rss/updates.xml, this feed provides the latest newly created releases for individual projects on PyPI, including the project name and description, release version, and a link to the release page. + +Project and release activity details +------------------------------------ + +PyPI publishes a "journal" of all project, package, and release +activity (including Owner and Maintainer additions and removals, and +source file and release additions and removals). You can query it with +a mix of :ref:`changelog-since` and :ref:`simple-api`. Call +``changelog_last_serial()`` (in :doc:`xml-rpc`) to get the current +revision of the journal (the last event's serial ID), then look at +``/simple/`` to get a list of all packages that currently +exist. Subsequently, you can call +``changelog_since_serial(since_serial)`` with the serial ID you +retrieved, and get the list of all actions that have occurred since +then. + +Example usage:: + + >>> import xmlrpc.client + >>> import arrow + >>> client = xmlrpc.client.ServerProxy('https://test.pypi.org/pypi') + >>> latefeb = arrow.get('2018-02-20 10:00:00') + >>> latefeb.timestamp + 1519120800 + >>> latefebstamp = latefeb.timestamp + >>> recentchanges = client.changelog(latefebstamp) + >>> len(recentchanges) + 7322 + >>> for entry in recentchanges: + ... if entry[0] == 'twine': + ... print(entry[1], " ", entry[3], " ", entry[2]) + ... + ... + ... + None add Owner brainwane 1519952529 + 1.10.0 add py2.py3 file twine-1.10.0-py2.py3-none-any.whl 1520023899 + 1.10.0 new release 1520023899 + 1.10.0rc1 add py2.py3 file twine-1.10.0rc1-py2.py3-none-any.whl 1520023900 + 1.10.0rc1 new release 1520023900 + 1.10.0rc1 add source file twine-1.10.0rc1.tar.gz 1520023902 + 1.10.0 add source file twine-1.10.0.tar.gz 1520023903 + 1.10.0 remove file twine-1.10.0.tar.gz 1520024758 + 1.10.0 remove file twine-1.10.0-py2.py3-none-any.whl 1520024797 + 1.10.0 remove 1520025270 + + +You could also request ``GET /simple/``, and record the ``ETag``, and +then periodically do a conditional HTTP GET to ``/simple/`` with that +ETag included. A 200 OK response indicates something has been added or +removed; if you get a 304 Not Modified, then nothing has changed. diff --git a/docs/api-reference/legacy.rst b/docs/api-reference/legacy.rst index e86056448a24..bf135b7987ce 100644 --- a/docs/api-reference/legacy.rst +++ b/docs/api-reference/legacy.rst @@ -4,6 +4,7 @@ Legacy API The "Legacy API" provides feature parity with `pypi-legacy`_, hence the term "legacy". +.. _simple-api: Simple Project API ------------------ diff --git a/docs/api-reference/xml-rpc.rst b/docs/api-reference/xml-rpc.rst index 168e74cd43ed..fd7726b04b8f 100644 --- a/docs/api-reference/xml-rpc.rst +++ b/docs/api-reference/xml-rpc.rst @@ -40,6 +40,11 @@ returned. ``release_data`` The `stable_version` flag is always an empty string. It was never fully supported anyway. +``release_downloads`` and ``top_packages`` No longer supported. Please +use `Google BigQuery +`_ +instead (`tips +`_). Package Querying ---------------- @@ -62,23 +67,21 @@ Package Querying Retrieve a list of `[role, package_name]` for a given `user`. Role is either `Maintainer` or `Owner`. -``release_downloads(package_name, release_version)`` - Retrieve a list of `[filename, download_count]` for a given `package_name` - and `release_version`. - ``release_urls(package_name, release_version)`` Retrieve a list of download URLs for the given `release_version`. Returns a list of dicts with the following keys: - * url - * packagetype ('sdist', 'bdist_wheel', etc) * filename - * size - * md5_digest - * downloads - * has_sig + * packagetype ('sdist', 'bdist_wheel', etc) * python_version (required version, or 'source', or 'any') + * size (an ``int``) + * md5_digest + * digests (a dict with two keys, "md5" and "sha256") + * has_sig (a boolean) + * upload_time (a ``DateTime`` object) * comment_text + * downloads (always says "-1") + * url ``release_data(package_name, release_version)`` Retrieve metadata describing a specific `release_version`. @@ -86,29 +89,35 @@ Package Querying * name * version - * stable_version (always an empty string) + * stable_version (always an empty string or None) + * bugtrack_url + * package_url + * release_url + * docs_url (URL of the packages.python.org docs if they've been supplied) + * home_page + * download_url + * project_url * author * author_email * maintainer * maintainer_email - * home_page - * license * summary - * description + * description (string, sometimes the entirety of a ``README``) + * license * keywords * platform - * download_url * classifiers (list of classifier strings) * requires * requires_dist * provides * provides_dist - * requires_external - * requires_python * obsoletes * obsoletes_dist - * project_url - * docs_url (URL of the packages.python.org docs if they've been supplied) + * requires_python + * requires_external + * _pypi_ordering + * _pypi_hidden + * downloads (``{'last_day': 0, 'last_week': 0, 'last_month': 0}``) If the release does not exist, an empty dictionary is returned. @@ -146,10 +155,6 @@ Package Querying the given classifiers. `classifiers` must be a list of Trove classifier strings. -``top_packages([number])`` - Retrieve the sorted list of packages ranked by number of downloads. - Optionally limit the list to the `number` given. - ``updated_releases(since)`` Retrieve a list of package releases made since the given timestamp. The releases will be listed in descending release date. @@ -159,22 +164,24 @@ Package Querying since the given timestamp. The packages will be listed in descending date of most recent change. +.. _changelog-since: Mirroring Support ----------------- ``changelog(since, with_ids=False)`` - Retrieve a list of `[name, version, timestamp, action]`, or - `[name, version, timestamp, action, id]` if `with_ids=True`, since the given - `since`. All `since` timestamps are UTC values. The argument is a UTC integer - seconds since the epoch. + Retrieve a list of `[name, version, timestamp, action]`, or `[name, + version, timestamp, action, id]` if `with_ids=True`, since the given + `since`. All `since` timestamps are UTC values. The argument is a + UTC integer seconds since the epoch (e.g., the ``timestamp`` method + to a ``datetime.datetime`` object). ``changelog_last_serial()`` - Retrieve the last event's serial id. + Retrieve the last event's serial id (an ``int``). ``changelog_since_serial(since_serial)`` Retrieve a list of `(name, version, timestamp, action, serial)` since the - event identified by the given `since_serial` All timestamps are UTC + event identified by the given ``since_serial``. All timestamps are UTC values. The argument is a UTC integer seconds since the epoch. ``list_packages_with_serial()`` diff --git a/docs/security.rst b/docs/security.rst index 786f3d59b1aa..d3bdae664c57 100644 --- a/docs/security.rst +++ b/docs/security.rst @@ -3,7 +3,14 @@ Security ======== +Security policy +--------------- To read the most up to date version of our security policy, please visit the application security page, available via the site_ footer. +Project and release activity details +------------------------------------ +Please see :doc:`api-reference/feeds` for how to track new and updated +releases on PyPI. + .. _site: https://pypi.org diff --git a/warehouse/templates/pages/help.html b/warehouse/templates/pages/help.html index 66bc52e640e4..42f7c86e47d8 100644 --- a/warehouse/templates/pages/help.html +++ b/warehouse/templates/pages/help.html @@ -38,6 +38,7 @@ {% macro private_indices() %}How can I publish my private packages to PyPI?{% endmacro %} {% macro admin_intervention() %}Why did my package or user registration get blocked?{% endmacro %} {% macro file_name_reuse() %}Why am I getting "File already exists" error?{% endmacro %} +{% macro APIs() %}Does PyPI have APIs I can use?{% endmacro %} {% block title %}Help{% endblock %} {% block content %} @@ -64,6 +65,7 @@

Common Questions

  • {{ private_indices() }}
  • {{ admin_intervention() }}
  • {{ file_name_reuse() }}
    • +
  • {{ APIs() }}
    • @@ -285,6 +287,12 @@

    {{ file_name_reuse() }}

    To avoid this situation, use Test PyPI to perform and check your upload first, before uploading to pypi.org.

    +
    +

    {{ APIs() }}

    +

    + Yes, including RSS feeds of new packages and new releases. Please see the API reference. +

    +