Ceph manual RGW storageclass region default

[Jiawei0227]

Hey I am very new to rook and I am trying to deploy Ceph in my cluster. Everything seems fine for block and file. Except when I was playing around RGW following the guide:

The storageclass has a field called region and it is set to us-east-1
I tried change this to something else and the bucket provisioning start to complain saying s3_bucket location constraint us-east-1 not found...

I am curious where this us-east-1 come from because I look at the code and it seems this is also the default value: https://github.com/rook/rook/blob/master/pkg/operator/ceph/object/s3-handlers.go#L48

Can someone help me to understand this? Thanks so much!

[travisn]

@thotz Can you take a look? I wouldn't expect the region name to be constant either. How can a different region be specified?

[thotz]

@Jiawei0227 can u please specify the rook version which you are using, I thought this was fixed by #8766

[Jiawei0227]

I was just changing the region field from us-east-1 to something else like us-west-1 and then the bucket fail to provision. So what I am curious is how does this get defined and how should I use it

[thotz]

You are using it in the right way in storageclass, there was a bug in an earlier release which got fixed in 1.7, can you please mention the rook version which you are using

[Jiawei0227]

Thanks! I think I am following the latest doc. The ceph version is ceph/ceph:v16.2.6
rook version seems to be v1.7.7

[Jiawei0227]

I double checked the version and it is 1.7.8

[Jiawei0227]

Gentle ping

[Jiawei0227]

Gentle ping. Can anyone help me to understand the region here? What does this mean?

[thotz]

Ur understanding correct region means the region of the bucket itself, even I am not sure why it is failing. Not sure whether any configuration is needed in the RGW side to make region work other than the default one

[Jiawei0227]

so this should in theory just be a random value that can be set to anything to represent the region or does it underlying do anything?

[thotz]

Here is my understanding after reading the RGW code a bit(I might be wrong), rgw admin ops or rest apis sets region to us-east-1 if it is not defined and also Rook uses this default value if the region is not defined in the storage class. Defining region value in storage class won't create a region in RGW backend, u need to create it. Storage class uses existing properties of the backend object store, does not define or create anything. I am a bit confusion how the region is defined in RGW, the official docs says it mapped zonegroup but from image it looks more like zone https://docs.ceph.com/en/latest/radosgw/multisite/#multi-site. So if u want to use us-west-1 as region then most probably u may need to create zone and zonegroups accordingly. Otherwise this operation can fail IMO

[Jiawei0227]

Thanks! This is very helpful!

[cbodley]

The storageclass has a field called region and it is set to us-east-1
I tried change this to something else and the bucket provisioning start to complain saying s3_bucket location constraint us-east-1 not found...

this 'location constraint' error sounds like it's coming from the CreateBucket request, based on what rook sends in the <LocationConstraint> field

rgw documents its expectations for this field in https://docs.ceph.com/en/latest/radosgw/placement/#s3-bucket-placement. basically, the location can either be just a region name (which rgw compares against its zonegroup api_names), or it can be 'region:placement', where 'placement' is the name of a placement target defined in the zonegroup

i'm not sure what the "storageclass" configuration is trying to do here. in rgw, each 'placement target' has a list of allowed storage classes. the bucket is created with a specific 'placement target' (which can be specified on creation via LocationConstraint), and the client can write objects to any of the 'storage classes' that placement target supports. so i think rook's configuration should ask for the placement target's name, instead of a storage class. in most cases they'll just leave it empty, so rook can omit the LocationConstraint part entirely, and rgw will use the default placement target defined in the zonegroup

and when it comes to the region name, i don't think rook should be requiring the user to fill that in at all. rook should which 'object store' it's operating on, and use the zonegroup api_name associated with that object store

[thotz]

@cbodley : The storageclass mentioned here is not what we have in ceph, it is defined for OBC(a Kubernetes object https://rook.io/docs/rook/v1.8/ceph-object-bucket-claim.html#storageclass). The rook set the region value into us-east-1 for the golang SDK if it is not defined in the storageclass of OBC. Here issue happened user-defined different value for region in storageclass Spec for the OBC without defining it as zonegroup in RGW, so the golang SDK was failing due to this request And you are right I guess golang SDK use this from session

[cbodley]

thanks for the link. i see that objectStoreName: my-store is one of the parameters. does rook have a way to look up the object store's zonegroup automatically? rook users may not even know what the object store chose for its zonegroup

[thotz]

@cbodley, Surely that makes sense and can be considered instead of setting default as us-east-1 if the region is not mentioned in storageclass of OBC.
Now what if the region value is mentioned, do we need to check whether zone groups exist then create the s3 client? What if the user mentioned us-east-1 this value will work even if zone group does not exist

[toabi]

I just ran into a similar issue, after upgrading from 1.7 to 1.8, OBCs didn't create buckets anymore with this error. I once wrote eu-central-1 in the storageclass because nowhere was mentioned where this was used and I thought it's more fitting for me.

I now recreated the storage class without the region and it started working again.

[thotz]

I guess this might be helpful #9754, please refer doc changes