Ability to set ceph dashboard rgw-api access/secret keys for multisite configuration

[jhoblitt]

What should the feature do:

Set/update the ceph rgw api credentials used by the dashboard to access rgw instances. The equivalent of:

$ ceph dashboard set-rgw-api-access-key -i /tmp/access-key
$ ceph dashboard set-rgw-api-secret-key -i /tmp/secret-key

Ideally, this would be controlled by the creation of a secret within the same namespace as the cephcluster. A secret per cephbobjectstore is not needed as the api credentials are "global" to the cluster. E.g.:

---
apiVersion: v1
kind: Secret
type: kubernetes.io/rook
metadata:
  name: rook-ceph-rgw-api-keys
  namespace: rook-ceph
data:
  access-key: <foo>
  secret-key: <bar>

What is use case behind this feature:

As of 1.10.1, enableRGWDashboard() first creates a system user and then sets the ceph dashboard rgw-api credentials to match that user. This is problematic for multisite setup of a non-master zone as 1) ceph, at least as of quincy, appears to be unable to hold credentials per rgw instance and the rgw-api credentials must be used to access all rgw instances in the ceph cluster and 2) a multisite zone has to be part of a realm and the realm replicates all users, including system users. This means that all rgw instances within a cluster need to to be accessible with the same system user credentials.

It is theoretically possible this could work if the non-master zone is the first rgw instance in a cluster, and the realm sync actually works before rook attempts to create the dashboard-admin user. However, after more than a week of testing, this has always resulted in failure to sync the credentials from the existing dashboard-admin user in the realm and rook never finishes zone configuration. Even if this is resolvable in terms of logic/order of operations changes, the ability to set/change the global rgw-api credentials is desirable within multisite in use.

I can not explain why the multisite integration test run under gha with two cephclusters within the same k8s cluster is working. I have only tested with two physically different k8s test clusters, which always results in rook failing at dashboard-admin user creation. I'm not sure if this is because the rgw is already in a realm in which the user already exists or because the realm/zonegroup is in a non-working state because the period was not committed.

2022-09-23 19:57:06.597754 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/zone-b-multisite-store". failed to create object store deployments: failed to start rgw health checker for CephObjectStore "rook-ceph/zone-b-multisite-store", will re-reconcile: failed to create bucket checker for CephObjectStore "rook-ceph/zone-b-multisite-store": failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "zone-b-multisite-store": failed to create s3 user. 2022-09-23T19:57:06.291+0000 7fa31de63500  0 period (45a13413-7e1f-4bb8-aeb1-f3ce1a56fa53 does not have zone a8919925-fa14-41e4-a9e1-8fdbd56dab79 configured

This means that in order for the dashboard to be able to access a non-master (or non-first zone in the realm/zonegroup) rgw instance something needs to change over the current behavior. There are at least these possible solutions:

1. Each cluster in the realm/zonegroups needs to be configurable to use a different name (id) for the dashboard-admin user to avoid collisions within the realm (I did not test hacking this into the operator).
2. Additional keys need to be added to an existing dashboard-admin user when a new zone is create (I tested adding multiple sets of keys to the dashboard-admin user after the zone was manually kicked into syncing).
3. Wait for ceph to add support for per rgw instance rgw-api credentials and then add this support to the operator.
4. The rgw-api credentials for each cluster within the realm/zonegroup have to be coordinated.

As all the users are replicated between rgw instances within the same realm, option \2 and \4 both result in the same set of keys being able to access all rgw instances. I think I slightly prefer option \4 as it results in only one set of credentials to need to consider / rotate.

Environment:

• rook 1.10.1
• ceph 17.2.3
• k8s 1.23.7

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[alimaredia]

unstale

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

[bumarcell]

any updates on this matter?

[travisn]

@thotz Thoughts on this?

[bumarcell]

BTW is there a way to set the endpoint for the manager, on which it talks to radosgw? or how is this info derived?
I can't even find where this config is stored 😓
I wonder why there's set-rgw-api-access-key and set-rgw-api-secret-key but nothing regarding host/port

[thotz]

@thotz Thoughts on this?

IMO, the user needs to be created only on the master zone, not all on zones so that the conflict can be avoided. @bumarcell please check the ceph dashboard devs about the how endpoint is figured.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[jhoblitt]

Do we want to keep this alive?

[bumarcell]

I'm not sure.. I was taken away by many other things and haven't come back to this again 🙈