[API feature request] Add package latest version publication date in /api/packages endpoint responses

[anlambert]

The /api/packages REST API endpoint of pub.dev enables to list all dart packages with their latest version info.
However, publication dates for latest package versions are missing from the API responses while they are available
when querying the /api/packages/{package_name} endpoint.

Would it be possible to add such info in /api/packages responses ?

Some context about this feature request, I work for Software Heritage, a non-profit organization
whose goal is to collect all publicly available software in source code form together with its development history,
replicate it massively to ensure its preservation , and share it with everyone who needs it.
We archive source code coming from VCS like git or svn but also from package managers (pypi, npm, ...).
Source code we have collected so far can be searched and browsed from that website.

We recently looked into archiving dart packages by developing a lister and loader whose roles are respectively to
enumerate all dart packages and load package tarballs content into our archive.
When listing packages from a package manager, we are interested to retrieve a last update date for a package as it
enables us to reload only packages with new versions since our last listing, which means less requests sent to your
REST API.

Currently, the only way to get the last update date for a dart package is to send an extra HTTP request
to the /api/packages/{package_name} endpoint for each package during the listing process.
This is not great as we are basically flooding your REST API with numerous requests to get such info.
So having latest package version publication dates info in the /api/packages responses would
be great to reduce the number of HTTP requests sent to your API by our lister.

The change to pub-dev source code to add such feature is pretty straightforward:

(swh) ✔ ~/dev/pub-dev [api-packages-add-published-date L|✚ 2] 
14:19 $ git diff
diff --git a/app/lib/frontend/handlers/custom_api.dart b/app/lib/frontend/handlers/custom_api.dart
index 8342cd1d..cb8551e4 100644
--- a/app/lib/frontend/handlers/custom_api.dart
+++ b/app/lib/frontend/handlers/custom_api.dart
@@ -175,6 +175,7 @@ Future<shelf.Response> apiPackagesHandler(shelf.Request request) async {
         'name': version.package,
         'latest': {
           'version': version.version,
+          'published': version.created!.toIso8601String(),
           'pubspec': version.pubspec!.asJson,
 
           // TODO: We should get rid of these:
diff --git a/app/test/frontend/handlers/custom_api_test.dart b/app/test/frontend/handlers/custom_api_test.dart
index e09768d6..392a2bce 100644
--- a/app/test/frontend/handlers/custom_api_test.dart
+++ b/app/test/frontend/handlers/custom_api_test.dart
@@ -28,6 +28,7 @@ void main() {
               'name': 'oxygen',
               'latest': {
                 'version': '1.2.0',
+                'published': isNotEmpty,
                 'pubspec': {
                   'name': 'oxygen',
                   'version': '1.2.0',
@@ -55,6 +56,7 @@ void main() {
               'name': 'flutter_titanium',
               'latest': {
                 'version': '1.10.0',
+                'published': isNotEmpty,
                 'pubspec': {
                   'name': 'flutter_titanium',
                   'version': '1.10.0',
@@ -84,6 +86,7 @@ void main() {
               'name': 'neon',
               'latest': {
                 'version': '1.0.0',
+                'published': isNotEmpty,
                 'pubspec': {
                   'name': 'neon',
                   'version': '1.0.0',

I can submit a PR if such feature request is accepted.

[isoos]

/api/packages is not an officially supported endpoint (not used by the pub client, and not listed on the https://pub.dev/help/api page. While we don't have an immediate plan to discontinue it, we wouldn't really want to encourage people using it, as there is a chance that in the future it is going away.

/api/packages/<package> is part of the official endpoints, it is heavily used and it is cached. We don't mind if you access these endpoints for all of the packages periodically.

For package name discovery, there is also /api/package-names, which is a more lightweight API than /api/packages. While it is not official either, I'd rather encourage using /api/package-names instead of the /api/packages.

/cc @jonasfj

[anlambert]

Thanks for your quick answer.

/api/packages/ is part of the official endpoints, it is heavily used and it is cached. We don't mind if you access these endpoints for all of the packages periodically.

That's good to know, we were worried about the performance impact on your side if we query that endpoint
for each package in a row (we have retry policy with exponential backoff for HTTP requests but still).

For package name discovery, there is also /api/package-names, which is a more lightweight API than /api/packages. While it is not official either, I'd rather encourage using /api/package-names instead of the /api/packages.

That's what we are currently using in our lister to get package names, we looked into the /api/packages
endpoint because it is returning latest package version info.

Closing this then.

[jonasfj]

@anlambert please consider using a custom user-agent with a link that tells us how to reach you, see:
https://github.com/dart-lang/pub/blob/master/doc/repository-spec-v2.md#metadata-headers

[anlambert]

@jonasfj All our listers use the following user-agent value: Software Heritage Lister (<version>).

If you need to reach us, you can use our contact form or create a task on our forge.

[jonasfj]

@anlambert consider including a link to the contact form or some page describing who you are and how to reach you in the user-agent.

That's really the best way to give people debugging logs a chance to reach out to you and help you.

[anlambert]

@jonasfj I modified our user agent value accordingly in this fresh commit.