S3 SELECT (Epic)

[jecluis]

Description

S3 SELECT provides relational-like operations, executed server-side, over data sets stored in objects. By executing operations on the objects server-side, the mechanism avoids having the user downloading potentially large objects to perform operations on a subset of a large data set.

Support for S3 SELECT already exists in RGW. This effort may have to be about testing, validating, and documenting how to use S3 SELECT in the context of s3gw.

This could open up using s3gw for AI/ML workloads.

Acceptance Criteria

• S3 SELECT works and the user is able to perform operations on large data sets, executed server-side
• The effort is properly tested
• There is supporting documentation about S3 SELECT and how to use it in the context of s3gw

Additional Information

• rgw documentation on S3 SELECT
• library used by rgw for S3 SELECT
• Amazon S3 SELECT documentation