[C++/Python] Add support for S3 Bucket Versioning

[asfimport]

Arrow offers a reasonably capable S3 interface, but it lacks support for S3 Buckets that have versioning enabled.  For information about what S3 bucket versioning is, see:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html

If Arrow is interacting with a bucket where versioning is enabled, there can be S3 keys that have multiple versions of content stored utilizing the same key name.  At the present moment, Arrow does not have the ability to:

1. Access versions of an S3 key rather than just the latest version of an S3 key.  There is no ability to specify the VersionId parameter of S3's GetObject API.

2. Report the VersionId created when a new S3 key is uploaded to a bucket.

   Along with S3, GCS also supports versioned buckets.

   https://cloud.google.com/storage/docs/object-versioning

   There are a few shortcomings of the Filesystem interface to support remote file systems that support versioning:

   1. The parameters for open_input_stream() and open_input_file() do not easily lend themselves to adding an additional parameter of "version" because they would be passed to all other implemented filesystems.  Most other file systems that exist don't actually support versioning.

   2. Upon completion of an S3 multipart upload (i.e., close() on an S3FileSystem output stream), there is not currently a way for the user to determine the VersionId or ETag of the S3 key that was created.  This is important to know because if there are multiple concurrent writers to S3, it should be possible to identify the written S3 key.

   Proposed solutions to enable S3 Bucket versioning:

   1. To allow library callers to read specific versions of an S3 key, extend only the S3FileSystem interface with two new API calls:

   open_input_stream_with_version()

   open_input_file_with_version()

   Both are like their namesakes from the normal FileSystem interface but take an additional parameter of a "version," which is a string representation of the VersionId returned by S3 when the S3 Key is created.  If these functions are called with an empty string for the specified version, the latest version of the S3 key will be returned.

   I'm a bit reluctant to create these specialized functions just on the S3FileSystem interface, but I also don't think it is appropriate to change open_input_stream() and open_input_file()'s parameter list for all filesystems just for functionality that is only implemented by a small number of filesystems.

   2. Allow callers to call ReadMetadata() on an S3FileSystem output stream to retrieve the metadata about the S3 key that has been written after the stream has been closed.  The metadata will likely include both a VersionId and a value for ETag.

Reporter: Rusty Conover

Related issues:

• [R] Access S3 files by version (is part of)

PRs and other links:

• GitHub Pull Request #13987

_{Note: This issue was originally created as ARROW-17544. Please see the migration documentation for further details.}

[asfimport]

Rusty Conover:
This is also related to ARROW-13895

[asfimport]

Antoine Pitrou / @pitrou:
Two notes:

1. I'm lukewarm about adding new method definitions, as it will result in combinatorial explosion (we also need async versions of these APIs as well as FileInfo-taking versions). One possibility would be to add the version as an additional optional argument (e.g.
   const util::optional<std::string>version& = {}).

2. GCS has both generation and metageneration, I'm not sure how to handle that nicely in the API. Should we just ignore the metageneration here? @coryan

[asfimport]

Carlos O'Ryan / @coryan:

2. GCS has both generation and metageneration, I'm not sure how to handle that nicely in the API. Should we just ignore the metageneration here?

I think you should ignore it for now, yes. metageneration changes when the metadata of an object changes, e.g., when you add new labels, or when you change the contentType attribute. And FYI, if object versioning is enabled, then each version has a different metadata and metageneration.

1. I'm lukewarm about adding new method definitions, ... (e.g. const util::optional<std::string>version& = {}).

It is not my place to design your API, just ignore me if this idea does not work for you. I would go further. You can expect additional optional parameters, e.g., GCS supports conditional operations ("read this object if its current generation is N" or "write this object if its current generation is M"), I believe S3 supports those too. If the objects are encrypted with customer-supplied encryption keys you need to provide them as part of each call. Some application want to override the retry and backoff policies for some operations.

This suggests you need something like a std::map<..., std::any>> options = {} as the extra parameter. This may serve as inspiration:

https://github.com/googleapis/google-cloud-cpp/blob/2b1e12f31ec49e55419f5740c406b707e08ca5e1/google/cloud/options.h#L89

[asfimport]

Antoine Pitrou / @pitrou:
Thanks for the feedback @coryan. I agree a general way to pass filesystem-specific options would be good.

We could use std::any (after we move to C++17) or we could perhaps have a base class InputStreamOptions that each filesystem might derive. Or we could have a std::variant, with the advantage that people won't pass arbitrary types by mistake.

@bkietz What is your opinion on std::any?

[asfimport]

Rusty Conover:
Great comments so far.

There is a deficiency in FileInfo right now because it doesn't contain the VersionId information.  Are we willing to extend it to contain this additional information?

There likely should be a way to specify this file information in a URI/path-based format, but I didn't want to extend the S3 URI parsing code to do it. S3 keys can be any set of Unicode characters < 1024 bytes, so I'm not sure it is valid to treat them as URIs where a ? character denotes the start of the query fields.  

https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html

It is important to be able to pass the VersionId or other request conditional information via the Python Dataset API; I'd like to be sure we're designing with that in mind.

[asfimport]

Antoine Pitrou / @pitrou:
No, we shouldn't treat filesystem paths as URIs. Paths and URIs are clearly distinct concepts, though unfortunately users often mix them up.

As for putting the version in FileInfo, that depends: does listing a pseudo-"directory" in S3 and GCS also gives each object's version? If so, then I agree it could go in FileInfo (note FileInfo's primary purpose is to list directory contents).

[asfimport]

Rusty Conover:
S3 splits out "listing" into two API methods.  One that is bucket versioning aware and one that isn't.

• List Objects V2 - https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html

  That doesn't include the "versions" of each S3 key. 

• List Object Versions - https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html

  That does list every version of every S3 key.

  Since there can be hundreds of versions of a single key, listing all versions of keys in a bucket can take much longer than just listing the bucket.

[asfimport]

Antoine Pitrou / @pitrou:

[ListObjectsV2] That doesn't include the "versions" of each S3 key.

Not even the version id of the latest version?

[ListObjectVersions] That does list every version of every S3 key.

Hmm, this looks similar to ListObjectsV2 with a couple unfortunate deviations (such as key-marker/version-id-marker vs. continuation-token).

So perhaps the version should be an optional component of FileInfo.

Then FileSelector could later get an optional flag to list all versions.

[asfimport]

Antoine Pitrou / @pitrou:
I'll note that GCS seems nicer since it doesn't have the ListObjectsV2 vs. ListObjectVersions distinction, you can use the same API for both:
https://cloud.google.com/storage/docs/json_api/v1/objects/list

[asfimport]

Rusty Conover:
Any thoughts on the attached PR?

[asfimport]

Antoine Pitrou / @pitrou:
Well, as I said we should avoid creating new methods for this.

[asfimport]

Rusty Conover:
I agree and wrote as much in the issue, but I'm unclear if there was a decision about expanding the open_input_stream() or open_input_file() parameter list.  

If we're waiting for C++17, what's the timeline for that?

I think if we're to implement InputStreamOptions with specific types per FileSystem type, that likely seems to be very reasonable and less prone to error by users.

What do you think about using ReadMetadata() to retrieve the uploaded S3 key information after the stream is closed?  That's also in the PR.

[asfimport]

Antoine Pitrou / @pitrou:

If we're waiting for C++17, what's the timeline for that?

It's being done in #13991

I think if we're to implement InputStreamOptions with specific types per FileSystem type, that likely seems to be very reasonable and less prone to error by users.

Probably, but it also means that options are not portable (for example, passing a version id would be different depending on the actual filesystem type, even though several different filesystem types would implement a version id).

We also must think about how to expose those options or parameters to Python.

What do you think about using ReadMetadata() to retrieve the uploaded S3 key information after the stream is closed? That's also in the PR.

That sounds a bit ad hoc. We should try to think more generally about which kind of information may be available when closing a file.

[asfimport]

Apache Arrow JIRA Bot:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned per project policy. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.