ceph_conf_overrides for rgw does not work properly

[eduardegorov]

Hello,

According to rgw-keystone-v3:

# The following variables should be added in your group_vars/rgws.yml file
# The double quotes are important, do NOT remove them.

ceph_conf_overrides:
  "client.rgw.{{ hostvars[inventory_hostname]['ansible_hostname'] }}":
     "rgw keystone api version": "3"
     "nss db path": "/var/lib/ceph/radosgw/ceph-radosgw.{{ ansible_hostname }}/nss"
      . . .

Unfortunately, this part of code seems to be evaluated on localhost (see 'local_action' part of roles/ceph-config/tasks/main.ym ) and ends up openstack-integration-related lines being included to the same "client.rgw.HOSTNAME" (usually just the first host in rgws group) section in ceph.conf for all rgw nodes. E.g., both on n1 and n2 nodes we would have the same content like:

. . .
[client.rgw.n3]
host = n3
keyring = /var/lib/ceph/radosgw/ceph-rgw.n3/keyring
log file = /var/log/ceph/ceph-rgw-n3.log
rgw frontends = civetweb port=x.x.x.x:xxxx num_threads=100

[client.rgw.n2]
host = n2
keyring = /var/lib/ceph/radosgw/ceph-rgw.n2/keyring
log file = /var/log/ceph/ceph-rgw-n2.log
rgw frontends = civetweb port=x.x.x.x:xxxx0 num_threads=100

[client.rgw.n1]
host = n1
keyring = /var/lib/ceph/radosgw/ceph-rgw.n1/keyring
log file = /var/log/ceph/ceph-rgw-n1.log
nss db path = /var/lib/ceph/radosgw/ceph-radosgw.n1/nss
rgw frontends = civetweb port=x.x.x.x:xxxx num_threads=100
rgw keystone accepted roles = Member, _member_, admin
rgw keystone admin domain = default
rgw keystone admin password = SuperPassword
rgw keystone admin project = admin
rgw keystone admin user = admin
rgw keystone api version = 3
rgw keystone revocation interval = 900
rgw keystone url = http://controller.url:123
rgw s3 auth use keystone = False
rgw_keystone_verify_ssl = False
. . .

1. Am I missing something? Is there any way to avoid this issue?
2. If not yet, you might be interested in a work-around we use for moving evaluation step to rgw nodes - config: add work-around for host-specific ceph_conf_overrides evaluation #2156

[guits]

@eduardegorov actually I think simply remove the run_once: true lines should be enough to fix your issue. I'll test this today and update your issue with my findings.

eg.

ceph-ansible/roles/ceph-config/tasks/main.yml

Line 23 in 4c2d98f

run_once: true

[guits]

update:
removing run_once: true won't be enough, it will overwrite the ceph.conf with the current host during iteration I guess.

[GillesMocellin]

On stable-3.1 branch, ovverride of rgw ceph config does still not work.
Here, it duplicates the client section :

[client.rgw.fidcllabs-oct-02]
host = fidcllabs-oct-02
keyring = /var/lib/ceph/radosgw/ceph-rgw.fidcllabs-oct-02/keyring
log file = /var/log/ceph/ceph-rgw-fidcllabs-oct-02.log
rgw frontends = civetweb port=10.203.40.102:8080 num_threads=100

[...]

[client.rgw.fidcllabs-oct-02]
host = fidcllabs-oct-02
keyring = /var/lib/ceph/radosgw/ceph-rgw.fidcllabs-oct-02/keyring
log file = /var/log/ceph/ceph-rgw-fidcllabs-oct-02.log
rgw frontends = civetweb port=10.203.40.102:8080 num_threads=100
rgw keystone accepted roles = Member, _member_, admin
rgw keystone admin domain = default
rgw keystone admin password = XXXXXXXXXXXX
rgw keystone admin project = service
rgw keystone admin user = ceph.rgw
rgw keystone api version = 3
rgw keystone revocation interval = 900
rgw keystone token cache size = 10000
rgw keystone url = http://osa-internal.labs.fidcl.cloud:35357
rgw s3 auth use keystone = true

And radosgw use only the first, so no keystone auth...

Also, cosmetic, but the [global] section is at the end.

[guits]

@GillesMocellin we have a BZ opened for this https://bugzilla.redhat.com/show_bug.cgi?id=1580408

this should be addressed soon by #2991

[guits]

Fixed by #2991