Adding support for ceph mgmt-gateway

[rkachach]

TODO

• review default SSL/TLS configuration
• unify cephadm root CA management
  (addressed by mgr/cephadm: adding mTLS for ceph mgmt-gateway and backend services communication #58402)
• fix prometheus/alertmanager access when secure_monitoring_stack is enabled
  (addressed by mgr/cephadm: adding mTLS for ceph mgmt-gateway and backend services communication #58402)
• fix the nginx image (now image from quay is used)

This pull request introduces a new design for Ceph applications based on a modular, service-based architecture. A new cephadm service mgmt-gateway based on nginx an open-source, high-performance web server known for its scalability, efficiency, and versatility. It will act as the new front-end and single point entry to the cluster, providing unified access to all Ceph applications, including the Dashboard and monitoring applications. In addition, Nginx enhances security and simplifies access management due to its robust community and high security standards.

Benefits of the new service

• Unified Access: Centralizing access through Nginx improves security and provide a single entry to the cluster mgmt.
• Improved user experience: User shouldn't care anymore about where each application is running (ip/host).
• High Availability for dashboard: Nginx HA mechanisms are used to provide high availability for ceph dashboard.
• High Availability for monitoring: Nginx HA mechanisms are used to provide high availability for monitoring.

High availability enhancements

The current cephadm/dashboard implementation lacks HA when it comes to monitoring services. Even when cephadm is able to deploy N instances of services such as grafana, prometheus or alertmanager when configuring the dashboard (using dashboard set-<service>-api-host API) it just picks the last configured daemon. In case this daemons goes down there's no automated fail-over to use redundant healthy instance. The following diagram reflects the current architecture (notice dashboard is configured to access directly the different monitoring services).

This problem is solved by using upstream HA features provided by nginx. The proposed solution makes sure of a dedicated internal server to act as rev-proxy for monitoring services. Dashboard is configured to use nginx end-points instead of using directly ip/host of the monitoring daemons. Following is a diagram of the new architecture:

As we can see in the above diagram, in the new architecture there are two servers:

• External server: this server is responsible of attending and routing external user requests. The idea is for this server is use it also for any extra processing we would like to perform for external users such as authentication, authorization, etc. This server relies on nginx upstream feature to group the monitoring applications (by category). HA mechanism is implemented by selecting one of the available healthy servers.

• Internal server: this server is responsible of attending and routing internal requests only. Similarly to the external case, this server relies on nginx upstream feature to provide monitoring HA this time for internal services. This server uses its own self-signed certificates to secure the communication with other internal clients.

Usage

cephadm:
ceph orch apply mgmt-gateway --placement=<your-destination-node>

Or by providing a detailed spec file (for custom certificates i.e):

service_type: mgmt-gateway
placement:
  hosts:
    - ceph-node-1
spec:
 port: 9443
 ssl_protocols:
   - TLSv1.2
   - TLSv1.3
 ssl_ciphers:
   - AES128-SHA
   - AES256-SHA
   - RC4-SHA
 ssl_certificate: |
   -----BEGIN CERTIFICATE-----
   < YOU CERT DATA HERE >
   -----END CERTIFICATE-----
 ssl_certificate_key: |
  -----BEGIN RSA PRIVATE KEY-----
   < YOU PRIV KEY DATA HERE >
  -----END RSA PRIVATE KEY-----

Example of the generated nginx config:

[root@ceph-node-0 ~]# cat /var/lib/ceph/bd861cc8-28ae-11ef-92e9-525400cfad85/mgmt-gateway.ceph-node-0/etc/nginx_external_server.conf 

server {
    listen                    443 ssl;
    listen                    [::]:443 ssl;
    ssl_certificate            /etc/nginx/ssl/nginx.crt;
    ssl_certificate_key /etc/nginx/ssl/nginx.key;
    ssl_protocols            TLSv1.2 TLSv1.3;
    # from:  https://ssl-config.mozilla.org/#server=nginx
    ssl_ciphers              ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305;

    # Only return Nginx in server header, no extra info will be provided
    server_tokens             off;

    # Perfect Forward Secrecy(PFS) is frequently compromised without this
    ssl_prefer_server_ciphers on;

    # Enable SSL session caching for improved performance
    ssl_session_tickets       off;
    ssl_session_timeout       1d;
    ssl_session_cache         shared:SSL:10m;

    # OCSP stapling
    ssl_stapling              on;
    ssl_stapling_verify       on;
    resolver_timeout 5s;

    # Security headers
    ## X-Content-Type-Options: avoid MIME type sniffing
    add_header X-Content-Type-Options nosniff;
    ## Strict Transport Security (HSTS): Yes
    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload";
    ## Enables the Cross-site scripting (XSS) filter in browsers.
    add_header X-XSS-Protection "1; mode=block";
    ## Content-Security-Policy (CSP): FIXME
    # add_header Content-Security-Policy "default-src 'self'; script-src 'self'; object-src 'none'; base-uri 'none'; require-trusted-types-for 'script'; frame-ancestors 'self';";



    location / {
        proxy_pass https://dashboard_servers;
        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
    }

    location /grafana {
        rewrite ^/grafana/(.*) /$1 break;
        proxy_pass https://grafana_servers;
    }

    location /prometheus {
        proxy_pass http://prometheus_servers;
    }

    location /alertmanager {
        proxy_pass http://alertmanager_servers;
    }
}

Contribution Guidelines

• To sign and title your commits, please refer to Submitting Patches to Ceph.

• If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

• When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

• Tracker (select at least one)
  • References tracker ticket
  • Very recent bug; references commit where it was introduced
  • New feature (ticket optional)
  • Doc update (no ticket needed)
  • Code cleanup (no ticket needed)
• Component impact
  • Affects Dashboard, opened tracker ticket
  • Affects Orchestrator, opened tracker ticket
  • No impact that needs to be tracked
• Documentation (select at least one)
  • Updates relevant documentation
  • No doc update is appropriate
• Tests (select at least one)
  • Includes unit test(s)
  • Includes integration test(s)
  • Includes bug reproducer
  • No tests

Show available Jenkins commands

• jenkins retest this please
• jenkins test classic perf
• jenkins test crimson perf
• jenkins test signed
• jenkins test make check
• jenkins test make check arm64
• jenkins test submodules
• jenkins test dashboard
• jenkins test dashboard cephadm
• jenkins test api
• jenkins test docs
• jenkins render docs
• jenkins test ceph-volume all
• jenkins test ceph-volume tox
• jenkins test windows
• jenkins test rook e2e

[phlogistonjohn]

I will not review the patch but I will take this opportunity to bike-shed the name a little bit: cluster-gateway doesn't make it clear that this is mainly for stuff like the dashboard and grafana, etc. ui-gateway might be clearer but too narrow. How about admin-gateway? Or management-gateway?

[rkachach]

I will not review the patch but I will take this opportunity to bike-shed the name a little bit: cluster-gateway doesn't make it clear that this is mainly for stuff like the dashboard and grafana, etc. ui-gateway might be clearer but too narrow. How about admin-gateway? Or management-gateway?

@phlogistonjohn no worries and the I think it's a good time to dicuss about the service name :)

The rev-proxy is includes monitoring stack (prometheus, alertmanager, grafana, ..). In addition, it can include any service app that we would run for cluster mgmt in the future. The ideal name I think could be "ingress" (to be aligned with k8s) but it's already in use as you know. I'm OK with going with another main as long as it describes better what's the purpose of the service 👍

[rkachach]

jenkins retest this please

[anthonyeleven]

Nice. Approval is for docs only. I've made a number of nitpicky wording requests.

[github-actions]

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

[rkachach]

jenkins test rook e2e

[rkachach]

jenkins test rook e2e

[adk3798]

Always tough to review PRs of this size, but given you already addressed my last set of review comments and nothing jumped out at me as an issue when checking over again, I approve

[adk3798]

https://pulpito.ceph.com/adking-2024-07-09_19:40:48-orch:cephadm-wip-adk-testing-2024-07-09-0905-distro-default-smithi/

Failures:

• mds_upgrade_sequence, known issue
• staggered upgrade with agent, known issue
• a couple in cluster logs type failures that don't look bad
• one failure in staggered upgrade that seemed to be a mismatch between what ceph versions and cephadm believed to be the version for a single RGW daemon. Needs investigation, but unrelated to PRs in the run
• thrash and rgw-ingress tests timing out, known issue

Generally a good run, nothing to block merging most of the PRs