external: add support for multisite in external cluster script

[parth-gr]

Add support for realm, zone and zonegroup

Description of your changes:

Which issue is resolved by this Pull Request:
Resolves # #11806

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide).
• Skip Tests for Docs: If this is only a documentation change, add the label skip-ci on the PR.
• Reviewed the developer guide on Submitting a Pull Request
• Pending release notes updated with breaking and/or notable changes for the next minor release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

[parth-gr]

@thotz Do we need to create multiple adminops user users as the configuration of users would be different? https://github.com/rook/rook/blob/master/deploy/examples/create-external-cluster-resources.py#L1064

For a normal rook-ceph cluster, I am not able to understand how this context is done for multisite, as I see we only create 1 admin ops user
#8772

I look we have two context used one into another objctx and opsctx,

rook/pkg/operator/ceph/object/controller.go

Line 233 in 805f1b1

objCtx, err := NewMultisiteContext(r.context, r.clusterInfo, cephObjectStore)

[travisn]

A couple questions

[travisn]

If no zone or zonegroup are specified, they will get the default of the empty string?

[BlaineEXE]

Suggest re-ordering these to be realm, then zonegroup, then zone, since that is the order of hierarchy.

[parth-gr]

Yaa, it will be "", I think that mean default

But will confirm

[parth-gr]

@thotz
What you will suggest doing here?
Either give default or leave as empty?

[thotz]

IMO giving default might be the right approach but only for zone and zonegroup. About empty strings I am not so sure, @cbodley ??

[alimaredia]

Just to rephrase my understand what's going on and the thread (in case Casey interjects): This python script is creating a radosgw user for future admin-api operations via the radosgw-admin command on an external Ceph cluster outside of K8s.

What you're wondering is if multisite is setup on the Ceph cluster you're creating the user on, what happens if you don't pass --rgw-realm & --rgw-zonegroup & --rgw-zone to the radosgw-admin command, or what happens if one of those values is empty.

[alimaredia]

I played around with some toy clusters with multisite enabled and not enabled and here's what I observed:

If a realm has been is created there is a default realm. This is different than the realm named default and the realm-id of this realm can be accessed via radosgw-admin realm get-default.

If no realms have been created there is no default realm set, but running radosgw-admin user create with --rgw-realm=default --rgw-zonegroup=default --rgw-zone=default is the same result as not having those flags for user creation.

Since the default realm is not the same as the realm named default in a multisite scenario, if the realm/zg/zone flags are not passed in or are set to empty values the users are created in the default realm but not in the realm default.
To be clear here is an example:
Say 2 rgw users U1, and U2 are created on a Ceph cluster.
Then realm1 is configured (with a zg and zone) with users U3 and U4 created in that realm. Since this realm is the first created it becomes the default realm.
Then a realm2 is configured with users U5 and U6 created in that realm.

Users U1 and U2 live in a realm called default.
If a new user U7 is created without multisite flags passed into the user create command it is created in the default realm, realm1.
If a new user U8 is created with multisite flags set to empty --rgw-realm= --rgw-zonegroup= --rgw-zone= passed into the user create command it is created in the default realm, realm1.
If a new user U9 is created with multisite flags set to the string default --rgw-realm=default --rgw-zonegroup=default --rgw-zone=default passed into the user create command it is created in the realm named default that contains users U1 and U2.

I'm not exactly sure what the right course of action is but I wanted to give everyone in this discussion this background information.

[parth-gr]

We just care about U7 and U8 for now, as I see they are the same so no need to worry,

If a new user U7 is created without multisite flags passed into the user create command it is created in the default realm, realm1.
If a new user U8 is created with multisite flags set to empty --rgw-realm= --rgw-zonegroup= --rgw-zone= passed into the user create command it is created in the default realm, realm1.

Thanks for the great explanation @alimaredia

[mergify]

This pull request has merge conflicts that must be resolved before it can be merged. @parth-gr please rebase it. https://rook.io/docs/rook/latest/Contributing/development-flow/#updating-your-fork

[thotz]

Since change is related to multisite, the multiple rgw servers will present and only one adminsop user is needed for the entire multisite. I am assuming user creation requests via admins op api need to redirect to the server in the master zone/zonegroup even if a different rgw server is used. But there was bug and @alimaredia was working to fix it a couple of months back and AFAIR it got fixed. So either in the create_rgw_admin_ops_user we need to check whether user exists or check it separately if exists then don't try to create the user again. But still, we may need to create k8s secret because the these rgw server can exist in different namespace

[parth-gr]

But still, we may need to create k8s secret because these rgw server can exist in different namespace

So the radosgw-admin admin ops access key and access secret should be the same, it is just that the k8s secret should be created in a different namespace,

That would be taken care of by ocs downstream and rook upstream.

SO I think this PR is good to go?

[parth-gr]

@thotz few doubts,

the multiple rgw servers will present and only one adminsop user is needed for the entire multisite.

One adminops user-> But with different zone there would be different acces secret created, right? SO why one?

multiple rgw servers -> It is just a namespace distinguish?

[parth-gr]

cbodley
Casey Bodley
Can you also have a look once

[parth-gr]

@thotz can you confirm this?
I see in the docs, we need to create --system user to facilitate authentication between daemons,
In this way i guess probably one system user is required,

https://docs.ceph.com/en/quincy/radosgw/multisite/#create-a-system-user
https://docs.ceph.com/en/quincy/radosgw/multisite/#migrating-a-single-site-deployment-to-multi-site

[thotz]

@thotz can you confirm this? I see in the docs, we need to create --system user to facilitate authentication between daemons, In this way i guess probably one system user is required,

https://docs.ceph.com/en/quincy/radosgw/multisite/#create-a-system-user https://docs.ceph.com/en/quincy/radosgw/multisite/#migrating-a-single-site-deployment-to-multi-site

For the external ceph cluster, this will already configured by the admin IMO, like script is not creating zone/zonegroup/realm etc. This is basic prerequisite for the multisite set up

[thotz]

@thotz few doubts,

the multiple rgw servers will present and only one adminsop user is needed for the entire multisite.

One adminops user-> But with different zone there would be different acces secret created, right? SO why one?

The user information will same across the multisite cluster, we are not creating new users in secondary zone/zonegroup. They are synced via realm pull request.

multiple rgw servers -> It is just a namespace distinguish?

AFAIR there is not hard rule for k8s service for RGW server existence, it can be in the same namespace, a different one or all together in different clusters.