[skip ci] adopt: import rgw ssl certificate into kv store

[guits]

Without this, when rgw is managed by cephadm, it fails to start because
the ssl certificate isn't present in the kv store.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1987010
Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1988404

Signed-off-by: Guillaume Abrioux gabrioux@redhat.com

[dsavineau]

There's multiple things to take in count here.

cephadm doesn't let you have a different ssl_certificate when deploying RGW nodes on multiple nodes with a single spec.

Let's say you have 3 nodes (cephaio-1, cephaio-2 and cephaio-3) with the rgws label and you run:

# ceph orch apply rgw object --port 8080 --ssl --placement="label:rgws"
Scheduled rgw.object update...

Then the ssl_certificate will always be fetch from config://rgw/cert/rgw.object for all nodes.

# ceph config dump|grep rgw_frontends
    client.rgw.object.cephaio-1.jinsps        basic     rgw_frontends                          beast ssl_port=8080 ssl_certificate=config://rgw/cert/rgw.object                           * 
    client.rgw.object.cephaio-2.dirkxl        basic     rgw_frontends                          beast ssl_port=8080 ssl_certificate=config://rgw/cert/rgw.object                           * 
    client.rgw.object.cephaio-3.lmiorw        basic     rgw_frontends                          beast ssl_port=8080 ssl_certificate=config://rgw/cert/rgw.object                           *

ceph-ansible has this flexibility but cephadm doesn't (at least without starting splitting the spec)

The only alternative would be to deploy RGW instance per host (even if the configuration across all nodes is the same).

# ceph orch apply rgw object1 --port 8080 --placement='cephaio-1' --ssl
Scheduled rgw.object1 update...  
# ceph orch apply rgw object2 --port 8080 --placement='cephaio-2' --ssl
Scheduled rgw.object2 update...
# ceph orch apply rgw object3 --port 8080 --placement='cephaio-3' --ssl
Scheduled rgw.object3 update...

And now we can set a ssl_certificate per host

# ceph config dump|grep rgw_frontends
    client.rgw.object1.cephaio-1.gijoio        basic     rgw_frontends                          beast ssl_port=8080 ssl_certificate=config://rgw/cert/rgw.object1                          * 
    client.rgw.object2.cephaio-2.buenep        basic     rgw_frontends                          beast ssl_port=8080 ssl_certificate=config://rgw/cert/rgw.object2                          * 
    client.rgw.object3.cephaio-3.nzcibt        basic     rgw_frontends                          beast ssl_port=8080 ssl_certificate=config://rgw/cert/rgw.object3                          *

@sebastian-philipp what do you think about this ?

I think a lot of RGW config from ceph-ansible are lost during the adoption since the client names aren't the same.

ceph-ansible uses : client.rgw.<hostname>.<instance_name>
cephadm uses : client.rgw.<service id>.<hostname>.<random 6 chars>

[sebastian-philipp]

There's multiple things to take in count here.

cephadm doesn't let you have a different ssl_certificate when deploying RGW nodes on multiple nodes with a single spec.

Let's say you have 3 nodes (cephaio-1, cephaio-2 and cephaio-3) with the rgws label and you run:

# ceph orch apply rgw object --port 8080 --ssl --placement="label:rgws"
Scheduled rgw.object update...

Then the ssl_certificate will always be fetch from config://rgw/cert/rgw.object for all nodes.

# ceph config dump|grep rgw_frontends
    client.rgw.object.cephaio-1.jinsps        basic     rgw_frontends                          beast ssl_port=8080 ssl_certificate=config://rgw/cert/rgw.object                           * 
    client.rgw.object.cephaio-2.dirkxl        basic     rgw_frontends                          beast ssl_port=8080 ssl_certificate=config://rgw/cert/rgw.object                           * 
    client.rgw.object.cephaio-3.lmiorw        basic     rgw_frontends                          beast ssl_port=8080 ssl_certificate=config://rgw/cert/rgw.object                           *

ceph-ansible has this flexibility but cephadm doesn't (at least without starting splitting the spec)

Right.

The only alternative would be to deploy RGW instance per host (even if the configuration across all nodes is the same).

# ceph orch apply rgw object1 --port 8080 --placement='cephaio-1' --ssl
Scheduled rgw.object1 update...  
# ceph orch apply rgw object2 --port 8080 --placement='cephaio-2' --ssl
Scheduled rgw.object2 update...
# ceph orch apply rgw object3 --port 8080 --placement='cephaio-3' --ssl
Scheduled rgw.object3 update...

And now we can set a ssl_certificate per host

I'd try to avoid it. Users are supposed to use haproxy for those deployments anyway. Having a dedicated SSL cert for different hapoxy backends is odd. I'd avoid it.

# ceph config dump|grep rgw_frontends
    client.rgw.object1.cephaio-1.gijoio        basic     rgw_frontends                          beast ssl_port=8080 ssl_certificate=config://rgw/cert/rgw.object1                          * 
    client.rgw.object2.cephaio-2.buenep        basic     rgw_frontends                          beast ssl_port=8080 ssl_certificate=config://rgw/cert/rgw.object2                          * 
    client.rgw.object3.cephaio-3.nzcibt        basic     rgw_frontends                          beast ssl_port=8080 ssl_certificate=config://rgw/cert/rgw.object3                          *

@sebastian-philipp what do you think about this ?

I think a lot of RGW config from ceph-ansible are lost during the adoption since the client names aren't the same.

ceph-ansible uses : client.rgw.<hostname>.<instance_name>
cephadm uses : client.rgw.<service id>.<hostname>.<random 6 chars>

That's true. Do you know which keys are set by ceph-ansible? Do you have a list? I mean, most of them should be covered by applying the spec already.

[dsavineau]

The only alternative would be to deploy RGW instance per host (even if the configuration across all nodes is the same).

# ceph orch apply rgw object1 --port 8080 --placement='cephaio-1' --ssl
Scheduled rgw.object1 update...  
# ceph orch apply rgw object2 --port 8080 --placement='cephaio-2' --ssl
Scheduled rgw.object2 update...
# ceph orch apply rgw object3 --port 8080 --placement='cephaio-3' --ssl
Scheduled rgw.object3 update...

And now we can set a ssl_certificate per host

I'd try to avoid it. Users are supposed to use haproxy for those deployments anyway. Having a dedicated SSL cert for different hapoxy backends is odd. I'd avoid it.

That's not odd at all and that's why I've opened https://tracker.ceph.com/issues/51972.

How do you managed RGW mutlisite replication with TLS then ? The zonegroup and zone needs to be created with all RGW endpoints. If you're using TLS you don't want to hidde the instances behind haproxy and keep unencrypted traffic between haproxy and the RGW instances.

Also it's pretty common to have multiple realm/zonegroup/zone on the same host and in that scenario you don't want a global load balancer for all instances but one for each realm/zonegroup/zone

FYI OpenStack Tripleo uses haproxy with TLS AND RGW nodes with TLS as part of their TLS everywhere policy. This setup is done part in TripleO (haproxy) and ceph-ansible (RGW with TLS). But if they need to migrate to ingress service with cephadm this won't work anymore. So it could be considered as a feature regression.

# ceph config dump|grep rgw_frontends
    client.rgw.object1.cephaio-1.gijoio        basic     rgw_frontends                          beast ssl_port=8080 ssl_certificate=config://rgw/cert/rgw.object1                          * 
    client.rgw.object2.cephaio-2.buenep        basic     rgw_frontends                          beast ssl_port=8080 ssl_certificate=config://rgw/cert/rgw.object2                          * 
    client.rgw.object3.cephaio-3.nzcibt        basic     rgw_frontends                          beast ssl_port=8080 ssl_certificate=config://rgw/cert/rgw.object3                          *

@sebastian-philipp what do you think about this ?
I think a lot of RGW config from ceph-ansible are lost during the adoption since the client names aren't the same.
ceph-ansible uses : client.rgw.<hostname>.<instance_name>
cephadm uses : client.rgw.<service id>.<hostname>.<random 6 chars>

That's true. Do you know which keys are set by ceph-ansible? Do you have a list? I mean, most of them should be covered by applying the spec already.

The spec only add rgw frontends and rgw_realm,rgw_zone when using multisite.

By default, we only have a few of them [1] mainly rgw frontends and rgw thread pool size (rgw_realm, rgw_zonegroup and rgw_zone for multisite setup).
But ceph-ansible is flexible enough via the ceph_config_override variable to add any key/value in the ceph configuration file.
So you can have any RGW option [2] if needed.
You have a look to some rgw keystone option (for the ceph-ansible TripleO CI job) [3].

[1] https://github.com/ceph/ceph-ansible/blob/master/roles/ceph-config/templates/ceph.conf.j2#L105-L125
[2] https://docs.ceph.com/en/latest/radosgw/config-ref/#ceph-object-gateway-config-reference
[3] https://github.com/ceph/ceph-ansible/blob/master/tests/functional/ooo-collocation/hosts#L8-L11

[dsavineau]

jenkins test centos-container-cephadm_adopt

[guits]

jenkins test centos-container-cephadm_adopt

[guits]

jenkins test centos-container-cephadm_adopt

[guits]

jenkins test centos-container-cephadm_adopt