rgw/s3vector: implement the VectorBucket API by yuvalif · Pull Request #66327 · ceph/ceph

yuvalif · 2025-11-19T20:23:50Z

This is a draft of adding Create/Get/List/Remove Vector Bucket APIs.
The idea is to borrow as much as possible from the existing mechanism for regular S3 buckets, so that in the future we will be able to support: metadata sync, policies, quota, stats etc. for vector buckets without the need to reimplement all of these mechanism.
Currently, the main difference between VectorBuckets and Buckets are with the prefixes of the RADOS object names. However, to make the code change less intrusive, I mainly used copy&paste, and inheritence to implement that
Once we have a working solution, refactoring MUST BE DONE so that the implementation is much smaller
- bucket_sobj refactoring
- RGWBucketCtl refactoring
- ~~sal::VectorBucket refactoring~~
- RGWRados refactoring
Bucket index code is removed from VectorBuckets
~~Currently the tests that verify combination of buckets and vector buckets are failing. if a bucket is created with the same name as a vector bucket, the bucket creation fails (and vice versa)~~

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins test classic perf Jenkins Job | Jenkins Job Definition
jenkins test crimson perf Jenkins Job | Jenkins Job Definition
jenkins test signed Jenkins Job | Jenkins Job Definition
jenkins test make check Jenkins Job | Jenkins Job Definition
jenkins test make check arm64 Jenkins Job | Jenkins Job Definition
jenkins test submodules Jenkins Job | Jenkins Job Definition
jenkins test dashboard Jenkins Job | Jenkins Job Definition
jenkins test dashboard cephadm Jenkins Job | Jenkins Job Definition
jenkins test api Jenkins Job | Jenkins Job Definition
jenkins test docs ReadTheDocs | Github Workflow Definition
jenkins test ceph-volume all Jenkins Jobs | Jenkins Jobs Definition
jenkins test windows Jenkins Job | Jenkins Job Definition
jenkins test rook e2e Jenkins Job | Jenkins Job Definition

You must only issue one Jenkins command per-comment. Jenkins does not understand
comments with more than one command.

according to: https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_S3_Vectors.html Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

do some fixes to message validation based on the tesst Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

cbodley · 2025-12-02T16:05:54Z

rgw/s3vector: disable mdlog writes for vector buckets
since the rest of the md sync code does not support them

TODO:

create RGWMetadataHandlers for vector bucket entrypoints (ex RGWBucketMetadataHandler) and instances (ex RGWBucketInstanceMetadataHandler)
attach them to the RGWMetadataManager in RGWCtl::init()

that's all that metadata sync should require, other than these mdlog entries

yuvalif · 2025-12-03T16:27:27Z

rgw/s3vector: disable mdlog writes for vector buckets
since the rest of the md sync code does not support them

TODO:

create RGWMetadataHandlers for vector bucket entrypoints (ex RGWBucketMetadataHandler) and instances (ex RGWBucketInstanceMetadataHandler)

attach them to the RGWMetadataManager in RGWCtl::init()

that's all that metadata sync should require, other than these mdlog entries

done here: 340d931
ran some basic testing - listing on 2ndary vector buckets created oin primary

cbodley

in the weekly meeting, we briefly discussed the contents of RGWBucketInfo::layout for vector buckets and i suggested marking them as "indexless"

cbodley · 2025-12-17T17:09:23Z

src/rgw/driver/rados/rgw_bucket.cc

+class RGWVectorBucketInstanceMetadataHandler : public RGWBucketInstanceMetadataHandler {
+protected:
+  int put_prepare(const DoutPrefixProvider* dpp, optional_yield y,
+                  const std::string& entry, RGWBucketCompleteInfo& bci,
+                  const std::optional<RGWBucketCompleteInfo>& old_bci,
+                  const RGWObjVersionTracker& objv_tracker,
+                  bool from_remote_zone) override { return 0;}


for multisite, RGWBucketInstanceMetadataHandler::put_prepare() is what calls init_default_bucket_layout() to initialize the local bucket layout

since you're overriding put_prepare(), you can just force indexless here:

// vector buckets are indexless bci.info.layout.current_index.layout.type = rgw::BucketIndexType::Indexless; return 0;

cbodley · 2025-12-17T17:09:49Z

src/rgw/rgw_rest_s3vector.cc

+    rgw::sal::VectorBucket::CreateParams createparams;
+    createparams.owner = s->user->get_id();
+    createparams.zonegroup_id = zonegroup.id;


then on creation, you can request indexless with:

// vector buckets are indexless createparams.index_type = rgw::BucketIndexType::Indexless;

yuvalif · 2025-12-25T11:11:41Z

jenkins test docs

should be merged after ceph#66327 Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

not implemented in this commit: * VectorBucket APIs (done in ceph#66327) * caching information as VectorBucket attributes and fetching them gtom there (after the VectoBucket PR is merged): schema, distance type * VectorBucket policy APIs * metadata support Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

adamemerson · 2026-01-13T18:29:20Z

src/rgw/rgw_sal.h

+                       const CreateParams& params,
+                       optional_yield y) = 0;
+
+    /** Get the cached attributes associated with this vector bucket */


Could we avoid using optional_yield since this is a new interface? Since we're trying to do async between rust and C++, I'd really rather avoid having the back-end have to deal with the possibility of blocking, and instead have that handled at the caller. (If the functions we call use take optional_yield they should be callable with the straight yield_context?

the vector bucket creation does not involve calling into lancedb, this is veriy similar to the flow of S3 bucket creation.
since optional_yield is used there, and since I'm using the same (or similar) calls, I think that we are "stuck" with it for now...

adamemerson · 2026-01-13T18:30:11Z

src/rgw/rgw_sal.h

+    /** Get the cached placement rule of this vector bucket */
+    virtual rgw_placement_rule& get_placement_rule() = 0;
+    /** Get the cached creation time of this vector bucket */
+    virtual ceph::real_time& get_creation_time() = 0;


Returning a non-const reference here doesn't seem right. SInce it's a 64-bit integer, could we just return it by value?

i copied that from sal::Bucket, but i completly agree that it should be by value

adamemerson · 2026-01-13T18:31:16Z

src/rgw/rgw_sal.h

+    /** Get the cached ID of this vector bucket */
+    virtual const std::string& get_bucket_id() const = 0;
+    /** Get the cached placement rule of this vector bucket */
+    virtual rgw_placement_rule& get_placement_rule() = 0;


This being a mutable reference doesn't seem right?

adamemerson · 2026-01-13T18:32:00Z

src/rgw/rgw_sal.h

+    /** Get the key for this vector bucket */
+    virtual rgw_bucket& get_key() = 0;
+    /** Get the info for this vector bucket */
+    virtual RGWBucketInfo& get_info() = 0;


adamemerson · 2026-01-13T21:22:05Z

src/rgw/driver/rados/rgw_rados.cc

+  ldpp_dout(dpp, 20) << "s3vector --- RGWRados::create_vector_bucke called" << dendl;
+  int ret = 0;
+
+#define MAX_CREATE_RETRIES 20 /* need to bound retries */


Please don't do this. Use a constexpr.

copied from...

adamemerson · 2026-01-13T21:27:10Z

src/rgw/driver/rados/rgw_rados.cc


 int RGWRados::get_raw_obj_ref(const DoutPrefixProvider *dpp, rgw_raw_obj obj, rgw_rados_ref* ref)
 {
+  ldpp_dout(dpp, 1) << "INFO: s3vector -- called RGWRados::get_raw_obj_ref()" << dendl;


This prints s3vector whether s3vector is involved or not?

will remove all of the debug log used to help with the call flow

adamemerson · 2026-01-13T21:29:57Z

src/rgw/driver/rados/rgw_rados.cc

+    RGWBucketEntryPoint ep;
+    r = ctl.vector_bucket->read_bucket_entrypoint_info(bucket_info.bucket,
+                                                &ep,
+						null_yield,


Why are we passing null_yield here?

copied from the s3 bucket code. will fix here, but i'm not sure why null_yield is used there? @cbodley ?

cbodley · 2026-01-21T16:51:27Z

src/rgw/rgw_sal.h

+    /** Store the cached bucket info into the backing store */
+    //virtual int put_info(const DoutPrefixProvider* dpp, bool exclusive, ceph::real_time mtime, optional_yield y) = 0;


you'll probably need this to implement things like PutVectorBucketPolicy to update existing bucket instance metadata objects

will add that. though the bucket policy part will be done in subsequent work

@cbodley added 2 APIs that will be used in setting vector bucket policies in: e69ecd4

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

yuvalif · 2026-01-28T09:14:33Z

cherry-picked into #66066

yuvalif added 2 commits October 27, 2025 10:55

rgw/s3vector: add stubs for API

58cce94

according to: https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_S3_Vectors.html Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/s3vector: add integration tests

ab3f119

do some fixes to message validation based on the tesst Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

yuvalif requested review from adamemerson, cbodley and soumyakoduri November 19, 2025 20:23

github-actions bot added rgw tests labels Nov 19, 2025

yuvalif mentioned this pull request Nov 20, 2025

rgw/s3vector: adding s3vectors API support to the RGW #66066

Open

34 tasks

yuvalif force-pushed the wip-vector-bucket-apis branch 2 times, most recently from 726f536 to 7e88e92 Compare November 27, 2025 10:50

yuvalif marked this pull request as ready for review November 27, 2025 15:39

yuvalif requested a review from a team as a code owner November 27, 2025 15:39

yuvalif requested a review from dang November 27, 2025 15:51

github-actions bot added the documentation label Dec 16, 2025

cbodley reviewed Dec 17, 2025

View reviewed changes

yuvalif requested a review from cbodley December 24, 2025 12:24

yuvalif added a commit to yuvalif/ceph that referenced this pull request Jan 1, 2026

rgw/s3vector: add lancedb support

6d1e6ef

should be merged after ceph#66327 Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

yuvalif added a commit to yuvalif/ceph that referenced this pull request Jan 1, 2026

rgw/s3vector: add lancedb support

a0f2896

should be merged after ceph#66327 Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

adamemerson approved these changes Jan 13, 2026

View reviewed changes

cbodley reviewed Jan 21, 2026

View reviewed changes

yuvalif requested a review from cbodley January 22, 2026 18:29

rgw/s3vector: implement the VectorBucket API

7e36888

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

yuvalif force-pushed the wip-vector-bucket-apis branch from e69ecd4 to 7e36888 Compare January 26, 2026 17:26

yuvalif added the needs-qa label Jan 26, 2026

yuvalif force-pushed the wip-s3vector branch from 0a1213d to 5ddc95a Compare January 28, 2026 09:13

yuvalif closed this Jan 28, 2026

		/** Store the cached bucket info into the backing store */
		//virtual int put_info(const DoutPrefixProvider* dpp, bool exclusive, ceph::real_time mtime, optional_yield y) = 0;

Conversation

yuvalif commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

cbodley commented Dec 2, 2025

Uh oh!

yuvalif commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cbodley left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuvalif commented Dec 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuvalif commented Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yuvalif commented Nov 19, 2025 •

edited

Loading

yuvalif commented Dec 3, 2025 •

edited

Loading

cbodley left a comment •

edited

Loading