New Datasource: S3 ListObjects

[neumantm]

Tell us more.

I want to let renovate manage the version of a Google-Cloud-CLI Download (See https://docs.cloud.google.com/sdk/docs/downloads-versioned-archives).
However, google has no standardized interface where versions could be listed and I didn't find any endpoint that can be made to work with a custom datasource.

I did however find this url:
https://storage.googleapis.com/cloud-sdk-release?prefix=google-cloud-cli-
It seems to behave like the ListObjects Endpoint of the S3 Protocol/API.

The response is an XML like this:

<ListBucketResult xmlns="http://doc.s3.amazonaws.com/2006-03-01">
    <Name>cloud-sdk-release</Name>
    <Prefix>google-cloud-cli</Prefix>
    <Marker/>
    <NextMarker>google-cloud-cli-469.0.0-darwin-arm.tar.gz</NextMarker>
    <IsTruncated>true</IsTruncated>
    <Contents>
      <Key>google-cloud-cli-371.0.0-darwin-arm.tar.gz</Key>
      <Generation>1643730046821449</Generation>
      <MetaGeneration>1</MetaGeneration>
      <LastModified>2022-02-01T15:40:46.853Z</LastModified>
      <ETag>"42de0b211d5f1bcad56cfaec1de736ca"</ETag>
      <Size>92564834</Size>
    </Contents>
    [...]
</ListBucketResult>

My suggestion is to add a datasource that can use such ListObjects endpoints, read all pages (using the NextMarker) and extract the versions from the Key field in the XML.
This datasource would then not be constraint to the single use-case I currently have, but have broader support for any situation where versions of a dependency are available in an S3 Bucket.

The only open question: How do we extract the version from the Key?
If anyone has suggestions, please comment below.

[neumantm]

On the topic of extracting the version from the Key. Here are my thoughts:

1. It is not possible to have users pass additional configuration options to a datasource, right?
   If that were somehow possible, users could just provide a regex pattern.
   a. We could (mis-)use the registry url to pass some simple configuration to the datasource. Similar to the deb datasource. Regex might be a bit much in an URL, but prefix and suffix to strip could work.
2. We could remove the prefix from the URL and any file extensions. In my example above the remainder would be 371.0.0-darwin-arm, which should be usable with docker versioning.