rook Failed to connect to the external object storage of the cluster

[kubecto]

Is this a bug report or feature request?

• Bug Report

Deviation from expected behavior:
According to https://rook.io/docs/rook/v1.10/Storage-Configuration/Object-Storage-RGW/object-storage/#connect-to-an-external-object-store here configuration, I tried to connect to a ceph cluster

ceph -s
  cluster.
    id: 6357d39c-cf19-4597-a950-c03bcfd3b138
    health: HEALTH_OK

  health: HEALTH_OK
    mon: 3 daemons, quorum ceph0,ceph1,ceph2 (age 11w)
    mgr: ceph0(active, since 3M), standbys: ceph2, ceph1
    mds: cephfs:1 k8s-cephfs:1 {cephfs:0=ceph1=up:active,k8s-cephfs:0=ceph2=up:active} 1 up:standby
    osd: 6 osds: 6 up (since 3M), 6 in (since 4M)
    rgw: 3 daemons active (ceph0.rgw0, ceph1.rgw0, ceph2.rgw0)

  task status.

  ceph1.rgw0, ceph2.rgw0
    pools: 14 pools, 304 pgs
    objects: 7.95k objects, 16 GiB
    usage: 340 GiB used, 560 GiB / 900 GiB avail
    pgs: 304 active+clean

[root@ceph0 ~]# ceph -v
ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable)

[root@ceph0 ~]# ceph config dump
WHO MASK LEVEL OPTION VALUE RO
  mon advanced auth_allow_insecure_global_id_reclaim false
  mgr advanced mgr/dashboard/ALERTMANAGER_API_HOST http://10.102.28.54:9093 *
  mgr advanced mgr/dashboard/GRAFANA_API_PASSWORD admin *
  mgr advanced mgr/dashboard/GRAFANA_API_SSL_VERIFY false *
  mgr advanced mgr/dashboard/GRAFANA_API_URL https://10.102.28.56:3000 *
  mgr advanced mgr/dashboard/GRAFANA_API_USERNAME admin *
  mgr advanced mgr/dashboard/PROMETHEUS_API_HOST http://10.102.28.54:9092 *
  mgr advanced mgr/dashboard/RGW_API_ACCESS_KEY 4SPM3BS0Q43Z41NAVIGM * mgr advanced mgr/dashboard/PROMETHEUS_API_HOST
  mgr advanced mgr/dashboard/RGW_API_HOST 10.102.28.54 * mgr advanced mgr/dashboard/RGW_API_ACCESS_KEY
  mgr advanced mgr/dashboard/RGW_API_PORT 8080 *
  mgr advanced mgr/dashboard/RGW_API_SCHEME http *
  mgr advanced mgr/dashboard/RGW_API_SECRET_KEY Da3sSFJ3mSNLftle1y30CBJBHbOxIrQrZzztu5AU *
  mgr advanced mgr/dashboard/RGW_API_USER_ID ceph-dashboard *
  mgr advanced mgr/dashboard/ceph0/server_addr 10.102.28.54 *
  mgr advanced mgr/dashboard/ceph1/server_addr 10.102.28.55 *
  mgr advanced mgr/dashboard/ceph2/server_addr 10.102.28.56 *
  mgr advanced mgr/dashboard/server_port 8443 *
  mgr advanced mgr/dashboard/ssl true *
  mgr advanced mgr/dashboard/ssl_server_port 8443 *

This is my connection configuration

# cat s3/external-store.yaml
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata: name: external-store
  name: external-store
  namespace: rook-ceph
spec.
  name: external-store namespace: rook-ceph spec.
    port: 8080
    externalRgwEndpoints.
      - ip: 10.102.28.54

If I try to follow the documentation

  healthCheck.
    bucket.
      enabled: true
      interval: 60s

The following error occurs, this field is not supported, so I removed it

strict decoding error: unknown field "spec.healthCheck.bucket.enabled"

Also, I can telnet to this address to make sure it's working.

telnet 10.102.28.54 8080
Trying 10.102.28.54...
Connected to 10.102.28.54.
Escape character is '^]'.

But after I try to run it, the operator container doesn't log anything important, it just cycles through the following error

2024-05-15 03:41:24.157692 I | ceph-spec: parsing mon endpoints: a=10.96.198.157:6789,b=10.96.242.212:6789,c=10.96.109.41:6789
2024-05-15 03:41:24.647860 I | ceph-object-controller: reconciling external object store
2024-05-15 03:41:24.650094 I | ceph-object-controller: skipping reconcile since operator is still initializing
2024-05-15 03:41:34.661386 I | ceph-spec: parsing mon endpoints: a=10.96.198.157:6789,b=10.96.242.212:6789,c=10.96.109.41:6789
2024-05-15 03:41:35.139167 I | ceph-object-controller: reconciling external object store
2024-05-15 03:41:35.141203 I | ceph-object-controller: skipping reconcile since operator is still initializing
2024-05-15 03:41:45.152401 I | ceph-spec: parsing mon endpoints: a=10.96.198.157:6789,b=10.96.242.212:6789,c=10.96.109.41:6789
2024-05-15 03:41:45.640873 I | ceph-object-controller: reconciling external object store
2024-05-15 03:41:45.642627 I | ceph-object-controller: skipping reconcile since operator is still initializing
2024-05-15 03:41:55.652139 I | ceph-spec: parsing mon endpoints: a=10.96.198.157:6789,b=10.96.242.212:6789,c=10.96.109.41:6789
2024-05-15 03:41:56.137383 I | ceph-object-controller: reconciling external object store
2024-05-15 03:41:56.139490 I | ceph-object-controller: skipping reconcile since operator is still initializing

When I try to delete it, it gets stuck

kubectl delete -f external-store.yaml
cephobjectstore.ceph.rook.io "external-store" deleted

When I restart a terminal, I find an error message from the operator

2024-05-15 04:08:31.937757 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/external-store". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: Secret "rgw-admin-ops-user" not found

My own feeling is that if you want to connect a ceph cluster, you need accesskey and secretAccessKey under normal circumstances, but the rook document does not specify this, I seem to think that this is the problem here, but how to solve it? In addition to

healthCheck:
bucket:
enabled: true
interval: 60s

Is the field unsupported bug affecting connections to external clusters? Can you help me explain?

I tried to create a secret and delete po operator connecting to rgw. Currently, there is no output in the log, only check

cat external-store.yaml
apiVersion: v1
kind: Secret
metadata:
  name: rgw-admin-ops-user
  namespace: rook-ceph
type: Opaque
data:
  accessKey: 4SPM3BS0Q43Z41NAVIGM
  secretKey: Da3sSFJ3mSNLftle1y30CBJBHbOxIrQrZzztu5AU
---
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: external-store
  namespace: rook-ceph
spec:
  gateway:
    port: 8080
    externalRgwEndpoints:
      - ip: 10.102.28.54

operator log

one
2024-05-15 06:16:04.474836 I | op-k8sutil: batch job rook-ceph-osd-prepare-node3 still exists
2024-05-15 06:16:07.475332 I | op-mgr: successful modules: dashboard
2024-05-15 06:16:07.476959 I | op-k8sutil: batch job rook-ceph-osd-prepare-node3 deleted
2024-05-15 06:16:07.482949 I | op-osd: started OSD provisioning job for node "node3"
2024-05-15 06:16:07.485012 I | op-osd: OSD orchestration status for node node1 is "completed"
2024-05-15 06:16:07.490284 I | op-osd: OSD orchestration status for node node2 is "completed"
2024-05-15 06:16:07.499671 I | op-osd: OSD orchestration status for node node3 is "starting"
2024-05-15 06:16:09.167302 I | op-osd: updating OSD 0 on node "node1"
2024-05-15 06:16:10.833841 I | op-osd: updating OSD 1 on node "node2"
2024-05-15 06:16:10.856418 I | op-osd: OSD orchestration status for node node3 is "orchestrating"
2024-05-15 06:16:10.856635 I | op-osd: OSD orchestration status for node node3 is "completed"
2024-05-15 06:16:12.501488 I | op-osd: updating OSD 2 on node "node3"
2024-05-15 06:16:13.409871 I | cephclient: successfully disallowed pre-quincy osds and enabled all new quincy-only functionality
2024-05-15 06:16:13.409896 I | op-osd: finished running OSDs in namespace "rook-ceph"
2024-05-15 06:16:13.409902 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph"
2024-05-15 06:16:13.421767 I | ceph-cluster-controller: reporting cluster telemetry
2024-05-15 06:16:18.738800 I | op-mon: checking if multiple mons are on the same node
2024-05-15 06:17:02.880879 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "external-store"
2024-05-15 06:17:02.911498 I | ceph-spec: parsing mon endpoints: a=10.96.198.157:6789,b=10.96.242.212:6789,c=10.96.109.41:6789
2024-05-15 06:17:03.385401 I | ceph-object-controller: reconciling external object store
2024-05-15 06:17:03.387265 I | ceph-object-controller: starting rgw health checker for CephObjectStore "rook-ceph/external-store"
^@^@

Expected behavior:

I hope that I can connect to external clusters normally, and the document needs to explain which versions of ceph clusters are supported to connect to, in addition, how to declare the connection authentication information or enter it in the CR field. At present, I am not sure how to connect to external clusters

How to reproduce it (minimal and precise):

Try using rook-1.10.12, when I am using k8s-1.28.6 cluster, and the external cluster is ceph version 14.2.22 from nautilus version, I feel like I need a key to connect to the external cluster in order to connect properly, However, these logs did not come out and the CR was deleted by the operator, nor could it be deleted normally. I was confused

Cluster Status to submit:

kubectl -n rook-ceph get cephcluster
NAME        DATADIRHOSTPATH   MONCOUNT   AGE   PHASE   MESSAGE                        HEALTH      EXTERNAL
rook-ceph   /var/lib/rook     3          78m   Ready   Cluster created successfully   HEALTH_OK
[root@k8s1 s3]# kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph -s
  cluster:
    id:     62390b1c-f43e-45fc-9963-8738ebf5762e
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum a,b,c (age 77m)
    mgr: a(active, since 76m), standbys: b
    osd: 3 osds: 3 up (since 77m), 3 in (since 77m)

  data:
    pools:   1 pools, 1 pgs
    objects: 2 objects, 449 KiB
    usage:   61 MiB used, 48 GiB / 48 GiB avail
    pgs:     1 active+clean

Environment:

• OS (e.g. from /etc/os-release):

# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel (e.g. uname -a):

# uname -a
Linux k8s1 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@k8s1 s3]#

• Cloud provider or hardware configuration:

In vsphere virtualization, each node has 4c8G and 16 GB data disks

• Rook version (use rook version inside of a Rook Pod):

rook-1.10.12

• Storage backend version (e.g. for ceph do ceph -v):

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph -v
ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)

• Kubernetes version (use kubectl version):

 kubectl version
Client Version: v1.28.6
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.6

• Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):

kubeadm

• Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph health
HEALTH_OK

[ehsan310]

What is your rook cluster manifest configuration ?

[kubecto]

cluster.yaml

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph # namespace:cluster
spec:
  cephVersion:
    # The container image used to launch the Ceph daemon pods (mon, mgr, osd, mds, rgw).
    # v16 is Pacific, and v17 is Quincy.
    # RECOMMENDATION: In production, use a specific version tag instead of the general v17 flag, which pulls the latest release and could result in different
    # versions running within the cluster. See tags available at https://hub.docker.com/r/ceph/ceph/tags/.
    # If you want to be more precise, you can always use a timestamp tag such quay.io/ceph/ceph:v17.2.3-20220805
    # This tag might not contain a new Ceph version, just security fixes from the underlying operating system, which will reduce vulnerabilities
    image: quay.io/ceph/ceph:v17.2.5
    # Whether to allow unsupported versions of Ceph. Currently `pacific` and `quincy` are supported.
    # Future versions such as `reef` (v18) would require this to be set to `true`.
    # Do not set to true in production.
    allowUnsupported: false
  # The path on the host where configuration files will be persisted. Must be specified.
  # Important: if you reinstall the cluster, make sure you delete this directory from each host or else the mons will fail to start on the new cluster.
  # In Minikube, the '/data' directory is configured to persist across reboots. Use "/data/rook" in Minikube environment.
  dataDirHostPath: /var/lib/rook
  # Whether or not upgrade should continue even if a check fails
  # This means Ceph's status could be degraded and we don't recommend upgrading but you might decide otherwise
  # Use at your OWN risk
  # To understand Rook's upgrade process of Ceph, read https://rook.io/docs/rook/latest/ceph-upgrade.html#ceph-version-upgrades
  skipUpgradeChecks: false
  # Whether or not continue if PGs are not clean during an upgrade
  continueUpgradeAfterChecksEvenIfNotHealthy: false
  # WaitTimeoutForHealthyOSDInMinutes defines the time (in minutes) the operator would wait before an OSD can be stopped for upgrade or restart.
  # If the timeout exceeds and OSD is not ok to stop, then the operator would skip upgrade for the current OSD and proceed with the next one
  # if `continueUpgradeAfterChecksEvenIfNotHealthy` is `false`. If `continueUpgradeAfterChecksEvenIfNotHealthy` is `true`, then operator would
  # continue with the upgrade of an OSD even if its not ok to stop after the timeout. This timeout won't be applied if `skipUpgradeChecks` is `true`.
  # The default wait timeout is 10 minutes.
  waitTimeoutForHealthyOSDInMinutes: 10
  mon:
    # Set the number of mons to be started. Generally recommended to be 3.
    # For highest availability, an odd number of mons should be specified.
    count: 3
    # The mons should be on unique nodes. For production, at least 3 nodes are recommended for this reason.
    # Mons should only be allowed on the same node for test environments where data loss is acceptable.
    allowMultiplePerNode: false
  mgr:
    # When higher availability of the mgr is needed, increase the count to 2.
    # In that case, one mgr will be active and one in standby. When Ceph updates which
    # mgr is active, Rook will update the mgr services to match the active mgr.
    count: 2
    allowMultiplePerNode: false
    modules:
      # Several modules should not need to be included in this list. The "dashboard" and "monitoring" modules
      # are already enabled by other settings in the cluster CR.
      - name: pg_autoscaler
        enabled: true
  # enable the ceph dashboard for viewing cluster status
  dashboard:
    enabled: true
    # serve the dashboard under a subpath (useful when you are accessing the dashboard via a reverse proxy)
    # urlPrefix: /ceph-dashboard
    # serve the dashboard at the given port.
    # port: 8443
    # serve the dashboard using SSL
    ssl: true
  # enable prometheus alerting for cluster
  monitoring:
    # requires Prometheus to be pre-installed
    enabled: false
  network:
    connections:
      # Whether to encrypt the data in transit across the wire to prevent eavesdropping the data on the network.
      # The default is false. When encryption is enabled, all communication between clients and Ceph daemons, or between Ceph daemons will be encrypted.
      # When encryption is not enabled, clients still establish a strong initial authentication and data integrity is still validated with a crc check.
      # IMPORTANT: Encryption requires the 5.11 kernel for the latest nbd and cephfs drivers. Alternatively for testing only,
      # you can set the "mounter: rbd-nbd" in the rbd storage class, or "mounter: fuse" in the cephfs storage class.
      # The nbd and fuse drivers are *not* recommended in production since restarting the csi driver pod will disconnect the volumes.
      encryption:
        enabled: false
      # Whether to compress the data in transit across the wire. The default is false.
      # Requires Ceph Quincy (v17) or newer. Also see the kernel requirements above for encryption.
      compression:
        enabled: false
    # enable host networking
    #provider: host
    # enable the Multus network provider
    #provider: multus
    #selectors:
      # The selector keys are required to be `public` and `cluster`.
      # Based on the configuration, the operator will do the following:
      #   1. if only the `public` selector key is specified both public_network and cluster_network Ceph settings will listen on that interface
      #   2. if both `public` and `cluster` selector keys are specified the first one will point to 'public_network' flag and the second one to 'cluster_network'
      #
      # In order to work, each selector value must match a NetworkAttachmentDefinition object in Multus
      #
      #public: public-conf --> NetworkAttachmentDefinition object name in Multus
      #cluster: cluster-conf --> NetworkAttachmentDefinition object name in Multus
    # Provide internet protocol version. IPv6, IPv4 or empty string are valid options. Empty string would mean IPv4
    #ipFamily: "IPv6"
    # Ceph daemons to listen on both IPv4 and Ipv6 networks
    #dualStack: false
  # enable the crash collector for ceph daemon crash collection
  crashCollector:
    disable: false
    # Uncomment daysToRetain to prune ceph crash entries older than the
    # specified number of days.
    #daysToRetain: 30
  # enable log collector, daemons will log on files and rotate
  logCollector:
    enabled: true
    periodicity: daily # one of: hourly, daily, weekly, monthly
    maxLogSize: 500M # SUFFIX may be 'M' or 'G'. Must be at least 1M.
  # automate [data cleanup process](https://github.com/rook/rook/blob/master/Documentation/Storage-Configuration/ceph-teardown.md#delete-the-data-on-hosts) in cluster destruction.
  cleanupPolicy:
    # Since cluster cleanup is destructive to data, confirmation is required.
    # To destroy all Rook data on hosts during uninstall, confirmation must be set to "yes-really-destroy-data".
    # This value should only be set when the cluster is about to be deleted. After the confirmation is set,
    # Rook will immediately stop configuring the cluster and only wait for the delete command.
    # If the empty string is set, Rook will not destroy any data on hosts during uninstall.
    confirmation: ""
    # sanitizeDisks represents settings for sanitizing OSD disks on cluster deletion
    sanitizeDisks:
      # method indicates if the entire disk should be sanitized or simply ceph's metadata
      # in both case, re-install is possible
      # possible choices are 'complete' or 'quick' (default)
      method: quick
      # dataSource indicate where to get random bytes from to write on the disk
      # possible choices are 'zero' (default) or 'random'
      # using random sources will consume entropy from the system and will take much more time then the zero source
      dataSource: zero
      # iteration overwrite N times instead of the default (1)
      # takes an integer value
      iteration: 1
    # allowUninstallWithVolumes defines how the uninstall should be performed
    # If set to true, cephCluster deletion does not wait for the PVs to be deleted.
    allowUninstallWithVolumes: false
  # To control where various services will be scheduled by kubernetes, use the placement configuration sections below.
  # The example under 'all' would have all services scheduled on kubernetes nodes labeled with 'role=storage-node' and
  # tolerate taints with a key of 'storage-node'.
  placement:
    all:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: node-role.kubernetes.io/storage-node
              operator: In
              values:
              - storage-node
      podAffinity:
      podAntiAffinity:
      topologySpreadConstraints:
      tolerations:
      - key: node-role.kubernetes.io/storage-node
        operator: Exists
# The above placement information can also be specified for mon, osd, and mgr components
#    mon:
# Monitor deployments may contain an anti-affinity rule for avoiding monitor
# collocation on the same node. This is a required rule when host network is used
# or when AllowMultiplePerNode is false. Otherwise this anti-affinity rule is a
# preferred rule with weight: 50.
#    osd:
#    prepareosd:
#    mgr:
#    cleanup:
  annotations:
#    all:
#    mon:
#    osd:
#    cleanup:
#    prepareosd:
# clusterMetadata annotations will be applied to only `rook-ceph-mon-endpoints` configmap and the `rook-ceph-mon` and `rook-ceph-admin-keyring` secrets.
# And clusterMetadata annotations will not be merged with `all` annotations.
#    clusterMetadata:
#       kubed.appscode.com/sync: "true"
# If no mgr annotations are set, prometheus scrape annotations will be set by default.
#    mgr:
  labels:
#    all:
#    mon:
#    osd:
#    cleanup:
#    mgr:
#    prepareosd:
# monitoring is a list of key-value pairs. It is injected into all the monitoring resources created by operator.
# These labels can be passed as LabelSelector to Prometheus
#    monitoring:
#    crashcollector:
  resources:
# The requests and limits set here, allow the mgr pod to use half of one CPU core and 1 gigabyte of memory
#    mgr:
#      limits:
#        cpu: "500m"
#        memory: "1024Mi"
#      requests:
#        cpu: "500m"
#        memory: "1024Mi"
# The above example requests/limits can also be added to the other components
#    mon:
#    osd:
# For OSD it also is a possible to specify requests/limits based on device class
#    osd-hdd:
#    osd-ssd:
#    osd-nvme:
#    prepareosd:
#    mgr-sidecar:
#    crashcollector:
#    logcollector:
#    cleanup:
  # The option to automatically remove OSDs that are out and are safe to destroy.
  removeOSDsIfOutAndSafeToRemove: false
  priorityClassNames:
    #all: rook-ceph-default-priority-class
    mon: system-node-critical
    osd: system-node-critical
    mgr: system-cluster-critical
    #crashcollector: rook-ceph-crashcollector-priority-class
  storage: # cluster level storage configuration and selection
    useAllNodes: true
    useAllDevices: true
    #deviceFilter:
    config:
      # crushRoot: "custom-root" # specify a non-default root label for the CRUSH map
      # metadataDevice: "md0" # specify a non-rotational storage so ceph-volume will use it as block db device of bluestore.
      # databaseSizeMB: "1024" # uncomment if the disks are smaller than 100 GB
      # journalSizeMB: "1024"  # uncomment if the disks are 20 GB or smaller
      # osdsPerDevice: "1" # this value can be overridden at the node or device level
      # encryptedDevice: "true" # the default value for this option is "false"
# Individual nodes and their config can be specified as well, but 'useAllNodes' above must be set to false. Then, only the named
# nodes below will be used as storage resources.  Each node's 'name' field should match their 'kubernetes.io/hostname' label.
    nodes:
      - name: "10.102.28.61"
        devices: # specific devices to use for storage can be specified for each node
          - name: "sdb"
      - name: "10.102.28.62"
        devices: # specific devices to use for storage can be specified for each node
          - name: "sdb"
      - name: "10.102.28.63"
        devices: # specific devices to use for storage can be specified for each node
          - name: "sdb"
    #       - name: "nvme01" # multiple osds can be created on high performance devices
    #         config:
    #           osdsPerDevice: "5"
    #       - name: "/dev/disk/by-id/ata-ST4000DM004-XXXX" # devices can be specified using full udev paths
    #     config: # configuration can be specified at the node level which overrides the cluster level config
    #   - name: "172.17.4.301"
    #     deviceFilter: "^sd."
    # when onlyApplyOSDPlacement is false, will merge both placement.All() and placement.osd
    onlyApplyOSDPlacement: false
  # The section for configuring management of daemon disruptions during upgrade or fencing.
  disruptionManagement:
    # If true, the operator will create and manage PodDisruptionBudgets for OSD, Mon, RGW, and MDS daemons. OSD PDBs are managed dynamically
    # via the strategy outlined in the [design](https://github.com/rook/rook/blob/master/design/ceph/ceph-managed-disruptionbudgets.md). The operator will
    # block eviction of OSDs by default and unblock them safely when drains are detected.
    managePodBudgets: true
    # A duration in minutes that determines how long an entire failureDomain like `region/zone/host` will be held in `noout` (in addition to the
    # default DOWN/OUT interval) when it is draining. This is only relevant when  `managePodBudgets` is `true`. The default value is `30` minutes.
    osdMaintenanceTimeout: 30
    # A duration in minutes that the operator will wait for the placement groups to become healthy (active+clean) after a drain was completed and OSDs came back up.
    # Operator will continue with the next drain if the timeout exceeds. It only works if `managePodBudgets` is `true`.
    # No values or 0 means that the operator will wait until the placement groups are healthy before unblocking the next drain.
    pgHealthCheckTimeout: 0
    # If true, the operator will create and manage MachineDisruptionBudgets to ensure OSDs are only fenced when the cluster is healthy.
    # Only available on OpenShift.
    manageMachineDisruptionBudgets: false
    # Namespace in which to watch for the MachineDisruptionBudgets.
    machineDisruptionBudgetNamespace: openshift-machine-api

  # healthChecks
  # Valid values for daemons are 'mon', 'osd', 'status'
  healthCheck:
    daemonHealth:
      mon:
        disabled: false
        interval: 45s
      osd:
        disabled: false
        interval: 60s
      status:
        disabled: false
        interval: 60s
    # Change pod liveness probe timing or threshold values. Works for all mon,mgr,osd daemons.
    livenessProbe:
      mon:
        disabled: false
      mgr:
        disabled: false
      osd:
        disabled: false
    # Change pod startup probe timing or threshold values. Works for all mon,mgr,osd daemons.
    startupProbe:
      mon:
        disabled: false
      mgr:
        disabled: false
      osd:
        disabled: false

operator.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: rook-ceph-operator-config
  # should be in the namespace of the operator
  namespace: rook-ceph # namespace:operator
data:
  # The logging level for the operator: ERROR | WARNING | INFO | DEBUG
  ROOK_LOG_LEVEL: "INFO"

  # Allow using loop devices for osds in test clusters.
  ROOK_CEPH_ALLOW_LOOP_DEVICES: "false"

  # Enable the CSI driver.
  # To run the non-default version of the CSI driver, see the override-able image properties in operator.yaml
  ROOK_CSI_ENABLE_CEPHFS: "true"
  # Enable the default version of the CSI RBD driver. To start another version of the CSI driver, see image properties below.
  ROOK_CSI_ENABLE_RBD: "true"
  # Enable the CSI NFS driver. To start another version of the CSI driver, see image properties below.
  ROOK_CSI_ENABLE_NFS: "false"
  ROOK_CSI_ENABLE_GRPC_METRICS: "false"

  # Set to true to enable Ceph CSI pvc encryption support.
  CSI_ENABLE_ENCRYPTION: "false"

  # Set to true to enable host networking for CSI CephFS and RBD nodeplugins. This may be necessary
  # in some network configurations where the SDN does not provide access to an external cluster or
  # there is significant drop in read/write performance.
  # CSI_ENABLE_HOST_NETWORK: "true"

  # Set to true to enable adding volume metadata on the CephFS subvolume and RBD images.
  # Not all users might be interested in getting volume/snapshot details as metadata on CephFS subvolume and RBD images.
  # Hence enable metadata is false by default.
  # CSI_ENABLE_METADATA: "true"

  # cluster name identifier to set as metadata on the CephFS subvolume and RBD images. This will be useful in cases
  # like for example, when two container orchestrator clusters (Kubernetes/OCP) are using a single ceph cluster.
  # CSI_CLUSTER_NAME: "my-prod-cluster"

  # Set logging level for cephCSI containers maintained by the cephCSI.
  # Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity.
  # CSI_LOG_LEVEL: "0"

  # Set logging level for Kubernetes-csi sidecar containers.
  # Supported values from 0 to 5. 0 for general useful logs (the default), 5 for trace level verbosity.
  # CSI_SIDECAR_LOG_LEVEL: "0"

  # Set replicas for csi provisioner deployment.
  CSI_PROVISIONER_REPLICAS: "2"

  # OMAP generator will generate the omap mapping between the PV name and the RBD image.
  # CSI_ENABLE_OMAP_GENERATOR need to be enabled when we are using rbd mirroring feature.
  # By default OMAP generator sidecar is deployed with CSI provisioner pod, to disable
  # it set it to false.
  # CSI_ENABLE_OMAP_GENERATOR: "false"

  # set to false to disable deployment of snapshotter container in CephFS provisioner pod.
  CSI_ENABLE_CEPHFS_SNAPSHOTTER: "true"

  # set to false to disable deployment of snapshotter container in NFS provisioner pod.
  CSI_ENABLE_NFS_SNAPSHOTTER: "true"

  # set to false to disable deployment of snapshotter container in RBD provisioner pod.
  CSI_ENABLE_RBD_SNAPSHOTTER: "true"

  # Enable cephfs kernel driver instead of ceph-fuse.
  # If you disable the kernel client, your application may be disrupted during upgrade.
  # See the upgrade guide: https://rook.io/docs/rook/latest/ceph-upgrade.html
  # NOTE! cephfs quota is not supported in kernel version < 4.17
  CSI_FORCE_CEPHFS_KERNEL_CLIENT: "true"

  # (Optional) policy for modifying a volume's ownership or permissions when the RBD PVC is being mounted.
  # supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html
  CSI_RBD_FSGROUPPOLICY: "File"

  # (Optional) policy for modifying a volume's ownership or permissions when the CephFS PVC is being mounted.
  # supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html
  CSI_CEPHFS_FSGROUPPOLICY: "File"

  # (Optional) policy for modifying a volume's ownership or permissions when the NFS PVC is being mounted.
  # supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html
  CSI_NFS_FSGROUPPOLICY: "File"

  # (Optional) Allow starting unsupported ceph-csi image
  ROOK_CSI_ALLOW_UNSUPPORTED_VERSION: "false"

  # (Optional) control the host mount of /etc/selinux for csi plugin pods.
  CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT: "false"

  # The default version of CSI supported by Rook will be started. To change the version
  # of the CSI driver to something other than what is officially supported, change
  # these images to the desired release of the CSI driver.
  ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.7.2"
  ROOK_CSI_REGISTRAR_IMAGE: "harbor.1stcs.cn/rook/csi-node-driver-registrar:v2.7.0"
  ROOK_CSI_RESIZER_IMAGE: "harbor.1stcs.cn/rook/csi-resizer:v1.7.0"
  ROOK_CSI_PROVISIONER_IMAGE: "harbor.1stcs.cn/rook/csi-provisioner:v3.4.0"
  ROOK_CSI_SNAPSHOTTER_IMAGE: "harbor.1stcs.cn/rook/csi-snapshotter:v6.2.1"
  ROOK_CSI_ATTACHER_IMAGE: "harbor.1stcs.cn/rook/csi-attacher:v4.1.0"

  # To indicate the image pull policy to be applied to all the containers in the csi driver pods.
  # ROOK_CSI_IMAGE_PULL_POLICY: "IfNotPresent"

  # (Optional) set user created priorityclassName for csi plugin pods.
  CSI_PLUGIN_PRIORITY_CLASSNAME: "system-node-critical"

  # (Optional) set user created priorityclassName for csi provisioner pods.
  CSI_PROVISIONER_PRIORITY_CLASSNAME: "system-cluster-critical"

  # CSI CephFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate.
  # Default value is RollingUpdate.
  # CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY: "OnDelete"
  # CSI RBD plugin daemonset update strategy, supported values are OnDelete and RollingUpdate.
  # Default value is RollingUpdate.
  # CSI_RBD_PLUGIN_UPDATE_STRATEGY: "OnDelete"
  # A maxUnavailable parameter of CSI RBD plugin daemonset update strategy.
  # Default value is 1.
  # CSI_RBD_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE: "1"

  # CSI NFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate.
  # Default value is RollingUpdate.
  # CSI_NFS_PLUGIN_UPDATE_STRATEGY: "OnDelete"

  # kubelet directory path, if kubelet configured to use other than /var/lib/kubelet path.
  # ROOK_CSI_KUBELET_DIR_PATH: "/var/lib/kubelet"

  # Labels to add to the CSI CephFS Deployments and DaemonSets Pods.
  # ROOK_CSI_CEPHFS_POD_LABELS: "key1=value1,key2=value2"
  # Labels to add to the CSI RBD Deployments and DaemonSets Pods.
  # ROOK_CSI_RBD_POD_LABELS: "key1=value1,key2=value2"
  # Labels to add to the CSI NFS Deployments and DaemonSets Pods.
  # ROOK_CSI_NFS_POD_LABELS: "key1=value1,key2=value2"

  # (Optional) CephCSI CephFS plugin Volumes
  # CSI_CEPHFS_PLUGIN_VOLUME: |
  #  - name: lib-modules
  #    hostPath:
  #      path: /run/current-system/kernel-modules/lib/modules/
  #  - name: host-nix
  #    hostPath:
  #      path: /nix

  # (Optional) CephCSI CephFS plugin Volume mounts
  # CSI_CEPHFS_PLUGIN_VOLUME_MOUNT: |
  #  - name: host-nix
  #    mountPath: /nix
  #    readOnly: true

  # (Optional) CephCSI RBD plugin Volumes
  # CSI_RBD_PLUGIN_VOLUME: |
  #  - name: lib-modules
  #    hostPath:
  #      path: /run/current-system/kernel-modules/lib/modules/
  #  - name: host-nix
  #    hostPath:
  #      path: /nix

  # (Optional) CephCSI RBD plugin Volume mounts
  # CSI_RBD_PLUGIN_VOLUME_MOUNT: |
  #  - name: host-nix
  #    mountPath: /nix
  #    readOnly: true

  # (Optional) CephCSI provisioner NodeAffinity (applied to both CephFS and RBD provisioner).
  CSI_PROVISIONER_NODE_AFFINITY: "node-role.kubernetes.io/storage-node=storage-node"
  # (Optional) CephCSI provisioner tolerations list(applied to both CephFS and RBD provisioner).
  # Put here list of taints you want to tolerate in YAML format.
  # CSI provisioner would be best to start on the same nodes as other ceph daemons.
  CSI_PROVISIONER_TOLERATIONS: |
    - key: "node-role.kubernetes.io/storage-node"
      operator: "Exists"
      effect: "NoSchedule"
  # (Optional) CephCSI plugin NodeAffinity (applied to both CephFS and RBD plugin).
  CSI_PLUGIN_NODE_AFFINITY: "node-role.kubernetes.io/storage-node=storage-node"
  # (Optional) CephCSI plugin tolerations list(applied to both CephFS and RBD plugin).
  # Put here list of taints you want to tolerate in YAML format.
  # CSI plugins need to be started on all the nodes where the clients need to mount the storage.
  CSI_PLUGIN_TOLERATIONS: |
    - key: "node-role.kubernetes.io/storage-node"
      operator: "Exists"
      effect: "NoSchedule"
  # (Optional) CephCSI RBD provisioner NodeAffinity (if specified, overrides CSI_PROVISIONER_NODE_AFFINITY).
  CSI_RBD_PROVISIONER_NODE_AFFINITY: "node-role.kubernetes.io/storage-node=storage-node"
  # (Optional) CephCSI RBD provisioner tolerations list(if specified, overrides CSI_PROVISIONER_TOLERATIONS).
  # Put here list of taints you want to tolerate in YAML format.
  # CSI provisioner would be best to start on the same nodes as other ceph daemons.
  CSI_RBD_PROVISIONER_TOLERATIONS: |
    - key: "node-role.kubernetes.io/storage-node"
      operator: "Exists"
      effect: "NoSchedule"
  # (Optional) CephCSI RBD plugin NodeAffinity (if specified, overrides CSI_PLUGIN_NODE_AFFINITY).
  CSI_RBD_PLUGIN_NODE_AFFINITY: "node-role.kubernetes.io/storage-node=storage-node"
  # (Optional) CephCSI RBD plugin tolerations list(if specified, overrides CSI_PLUGIN_TOLERATIONS).
  # Put here list of taints you want to tolerate in YAML format.
  # CSI plugins need to be started on all the nodes where the clients need to mount the storage.
  CSI_RBD_PLUGIN_TOLERATIONS: |
    - key: "node-role.kubernetes.io/storage-node"
      operator: "Exists"
      effect: "NoSchedule"
  # (Optional) CephCSI CephFS provisioner NodeAffinity (if specified, overrides CSI_PROVISIONER_NODE_AFFINITY).
  CSI_CEPHFS_PROVISIONER_NODE_AFFINITY: "node-role.kubernetes.io/storage-node=storage-node"
  # (Optional) CephCSI CephFS provisioner tolerations list(if specified, overrides CSI_PROVISIONER_TOLERATIONS).
  # Put here list of taints you want to tolerate in YAML format.
  # CSI provisioner would be best to start on the same nodes as other ceph daemons.
  CSI_CEPHFS_PROVISIONER_TOLERATIONS: |
    - key: "node-role.kubernetes.io/storage-node"
      operator: "Exists"
      effect: "NoSchedule"
  # (Optional) CephCSI CephFS plugin NodeAffinity (if specified, overrides CSI_PLUGIN_NODE_AFFINITY).
  CSI_CEPHFS_PLUGIN_NODE_AFFINITY: "node-role.kubernetes.io/storage-node=storage-node"
  # NOTE: Support for defining NodeAffinity for operators other than "In" and "Exists" requires the user to input a
  # valid v1.NodeAffinity JSON or YAML string. For example, the following is valid YAML v1.NodeAffinity:
  # CSI_CEPHFS_PLUGIN_NODE_AFFINITY: |
  #   requiredDuringSchedulingIgnoredDuringExecution:
  #     nodeSelectorTerms:
  #       - matchExpressions:
  #         - key: myKey
  #           operator: DoesNotExist
  # (Optional) CephCSI CephFS plugin tolerations list(if specified, overrides CSI_PLUGIN_TOLERATIONS).
  # Put here list of taints you want to tolerate in YAML format.
  # CSI plugins need to be started on all the nodes where the clients need to mount the storage.
  CSI_CEPHFS_PLUGIN_TOLERATIONS: |
    - key: "node-role.kubernetes.io/storage-node"
      operator: "Exists"
      effect: "NoSchedule"
  # (Optional) CephCSI NFS provisioner NodeAffinity (overrides CSI_PROVISIONER_NODE_AFFINITY).
  CSI_NFS_PROVISIONER_NODE_AFFINITY: "node-role.kubernetes.io/storage-node=storage-node"
  # (Optional) CephCSI NFS provisioner tolerations list (overrides CSI_PROVISIONER_TOLERATIONS).
  # Put here list of taints you want to tolerate in YAML format.
  # CSI provisioner would be best to start on the same nodes as other ceph daemons.
  CSI_NFS_PROVISIONER_TOLERATIONS: |
    - key: "node-role.kubernetes.io/storage-node"
      operator: "Exists"
      effect: "NoSchedule"
  # (Optional) CephCSI NFS plugin NodeAffinity (overrides CSI_PLUGIN_NODE_AFFINITY).
  CSI_NFS_PLUGIN_NODE_AFFINITY: "node-role.kubernetes.io/storage-node=storage-node"
  # (Optional) CephCSI NFS plugin tolerations list (overrides CSI_PLUGIN_TOLERATIONS).
  # Put here list of taints you want to tolerate in YAML format.
  # CSI plugins need to be started on all the nodes where the clients need to mount the storage.
  CSI_NFS_PLUGIN_TOLERATIONS: |
    - key: "node-role.kubernetes.io/storage-node"
      operator: "Exists"
      effect: "NoSchedule"
  # (Optional) CEPH CSI RBD provisioner resource requirement list, Put here list of resource
  # requests and limits you want to apply for provisioner pod
  #CSI_RBD_PROVISIONER_RESOURCE: |
  #  - name : csi-provisioner
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  #  - name : csi-resizer
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  #  - name : csi-attacher
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  #  - name : csi-snapshotter
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  #  - name : csi-rbdplugin
  #    resource:
  #      requests:
  #        memory: 512Mi
  #        cpu: 250m
  #      limits:
  #        memory: 1Gi
  #        cpu: 500m
  #  - name : csi-omap-generator
  #    resource:
  #      requests:
  #        memory: 512Mi
  #        cpu: 250m
  #      limits:
  #        memory: 1Gi
  #        cpu: 500m
  #  - name : liveness-prometheus
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 50m
  #      limits:
  #        memory: 256Mi
  #        cpu: 100m
  # (Optional) CEPH CSI RBD plugin resource requirement list, Put here list of resource
  # requests and limits you want to apply for plugin pod
  #CSI_RBD_PLUGIN_RESOURCE: |
  #  - name : driver-registrar
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 50m
  #      limits:
  #        memory: 256Mi
  #        cpu: 100m
  #  - name : csi-rbdplugin
  #    resource:
  #      requests:
  #        memory: 512Mi
  #        cpu: 250m
  #      limits:
  #        memory: 1Gi
  #        cpu: 500m
  #  - name : liveness-prometheus
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 50m
  #      limits:
  #        memory: 256Mi
  #        cpu: 100m
  # (Optional) CEPH CSI CephFS provisioner resource requirement list, Put here list of resource
  # requests and limits you want to apply for provisioner pod
  #CSI_CEPHFS_PROVISIONER_RESOURCE: |
  #  - name : csi-provisioner
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  #  - name : csi-resizer
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  #  - name : csi-attacher
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  #  - name : csi-snapshotter
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  #  - name : csi-cephfsplugin
  #    resource:
  #      requests:
  #        memory: 512Mi
  #        cpu: 250m
  #      limits:
  #        memory: 1Gi
  #        cpu: 500m
  #  - name : liveness-prometheus
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 50m
  #      limits:
  #        memory: 256Mi
  #        cpu: 100m
  # (Optional) CEPH CSI CephFS plugin resource requirement list, Put here list of resource
  # requests and limits you want to apply for plugin pod
  #CSI_CEPHFS_PLUGIN_RESOURCE: |
  #  - name : driver-registrar
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 50m
  #      limits:
  #        memory: 256Mi
  #        cpu: 100m
  #  - name : csi-cephfsplugin
  #    resource:
  #      requests:
  #        memory: 512Mi
  #        cpu: 250m
  #      limits:
  #        memory: 1Gi
  #        cpu: 500m
  #  - name : liveness-prometheus
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 50m
  #      limits:
  #        memory: 256Mi
  #        cpu: 100m

  # (Optional) CEPH CSI NFS provisioner resource requirement list, Put here list of resource
  # requests and limits you want to apply for provisioner pod
  # CSI_NFS_PROVISIONER_RESOURCE: |
  #  - name : csi-provisioner
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  #  - name : csi-nfsplugin
  #    resource:
  #      requests:
  #        memory: 512Mi
  #        cpu: 250m
  #      limits:
  #        memory: 1Gi
  #        cpu: 500m
  # (Optional) CEPH CSI NFS plugin resource requirement list, Put here list of resource
  # requests and limits you want to apply for plugin pod
  # CSI_NFS_PLUGIN_RESOURCE: |
  #  - name : driver-registrar
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 50m
  #      limits:
  #        memory: 256Mi
  #        cpu: 100m
  #  - name : csi-nfsplugin
  #    resource:
  #      requests:
  #        memory: 512Mi
  #        cpu: 250m
  #      limits:
  #        memory: 1Gi
  #        cpu: 500m

  # Configure CSI Ceph FS grpc and liveness metrics port
  # Set to true to enable Ceph CSI liveness container.
  CSI_ENABLE_LIVENESS: "false"
  # CSI_CEPHFS_GRPC_METRICS_PORT: "9091"
  # CSI_CEPHFS_LIVENESS_METRICS_PORT: "9081"
  # Configure CSI RBD grpc and liveness metrics port
  # CSI_RBD_GRPC_METRICS_PORT: "9090"
  # CSI_RBD_LIVENESS_METRICS_PORT: "9080"
  # CSIADDONS_PORT: "9070"

  # Whether the OBC provisioner should watch on the operator namespace or not, if not the namespace of the cluster will be used
  ROOK_OBC_WATCH_OPERATOR_NAMESPACE: "true"

  # Whether to start the discovery daemon to watch for raw storage devices on nodes in the cluster.
  # This daemon does not need to run if you are only going to create your OSDs based on StorageClassDeviceSets with PVCs.
  ROOK_ENABLE_DISCOVERY_DAEMON: "false"
  # The timeout value (in seconds) of Ceph commands. It should be >= 1. If this variable is not set or is an invalid value, it's default to 15.
  ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS: "15"
  # Enable the csi addons sidecar.
  CSI_ENABLE_CSIADDONS: "false"
  # ROOK_CSIADDONS_IMAGE: "quay.io/csiaddons/k8s-sidecar:v0.5.0"
  # The CSI GRPC timeout value (in seconds). It should be >= 120. If this variable is not set or is an invalid value, it's default to 150.
  CSI_GRPC_TIMEOUT_SECONDS: "150"

  ROOK_DISABLE_ADMISSION_CONTROLLER: "true"

  # Enable topology based provisioning.
  CSI_ENABLE_TOPOLOGY: "false"
  # Domain labels define which node labels to use as domains
  # for CSI nodeplugins to advertise their domains
  # NOTE: the value here serves as an example and needs to be
  # updated with node labels that define domains of interest
  # CSI_TOPOLOGY_DOMAIN_LABELS: "kubernetes.io/hostname,topology.kubernetes.io/zone,topology.rook.io/rack"
---
# OLM: BEGIN OPERATOR DEPLOYMENT
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rook-ceph-operator
  namespace: rook-ceph # namespace:operator
  labels:
    operator: rook
    storage-backend: ceph
    app.kubernetes.io/name: rook-ceph
    app.kubernetes.io/instance: rook-ceph
    app.kubernetes.io/component: rook-ceph-operator
    app.kubernetes.io/part-of: rook-ceph-operator
spec:
  selector:
    matchLabels:
      app: rook-ceph-operator
  strategy:
    type: Recreate
  replicas: 1
  template:
    metadata:
      labels:
        app: rook-ceph-operator
    spec:
      tolerations:
      - key: "node-role.kubernetes.io/storage-node"
        operator: "Exists"
        effect: "NoSchedule"
      serviceAccountName: rook-ceph-system
      containers:
        - name: rook-ceph-operator
          image: rook/ceph:v1.10.12
          args: ["ceph", "operator"]
          securityContext:
            runAsNonRoot: true
            runAsUser: 2016
            runAsGroup: 2016
          volumeMounts:
            - mountPath: /var/lib/rook
              name: rook-config
            - mountPath: /etc/ceph
              name: default-config-dir
            - mountPath: /etc/webhook
              name: webhook-cert
          ports:
            - containerPort: 9443
              name: https-webhook
              protocol: TCP
          env:
            # If the operator should only watch for cluster CRDs in the same namespace, set this to "true".
            # If this is not set to true, the operator will watch for cluster CRDs in all namespaces.
            - name: ROOK_CURRENT_NAMESPACE_ONLY
              value: "false"
            # Rook Discover toleration. Will tolerate all taints with all keys.
            # Choose between NoSchedule, PreferNoSchedule and NoExecute:
            # - name: DISCOVER_TOLERATION
            #   value: "NoSchedule"
            # (Optional) Rook Discover toleration key. Set this to the key of the taint you want to tolerate
            # - name: DISCOVER_TOLERATION_KEY
            #   value: "<KeyOfTheTaintToTolerate>"
            # (Optional) Rook Discover tolerations list. Put here list of taints you want to tolerate in YAML format.
            - name: DISCOVER_TOLERATIONS
              value: |
                - key: "node-role.kubernetes.io/storage-node"
                  operator: "Exists"
                  effect: "NoSchedule"
            # (Optional) Rook Discover priority class name to set on the pod(s)
            # - name: DISCOVER_PRIORITY_CLASS_NAME
            #   value: "<PriorityClassName>"
            # (Optional) Discover Agent NodeAffinity.
            - name: DISCOVER_AGENT_NODE_AFFINITY
              value: "node-role.kubernetes.io/storage-node=storage-node"
            # (Optional) Discover Agent Pod Labels.
            # - name: DISCOVER_AGENT_POD_LABELS
            #   value: "key1=value1,key2=value2"

            # The duration between discovering devices in the rook-discover daemonset.
            - name: ROOK_DISCOVER_DEVICES_INTERVAL
              value: "60m"

            # Whether to start pods as privileged that mount a host path, which includes the Ceph mon and osd pods.
            # Set this to true if SELinux is enabled (e.g. OpenShift) to workaround the anyuid issues.
            # For more details see https://github.com/rook/rook/issues/1314#issuecomment-355799641
            - name: ROOK_HOSTPATH_REQUIRES_PRIVILEGED
              value: "false"

            # Disable automatic orchestration when new devices are discovered
            - name: ROOK_DISABLE_DEVICE_HOTPLUG
              value: "false"

            # Provide customised regex as the values using comma. For eg. regex for rbd based volume, value will be like "(?i)rbd[0-9]+".
            # In case of more than one regex, use comma to separate between them.
            # Default regex will be "(?i)dm-[0-9]+,(?i)rbd[0-9]+,(?i)nbd[0-9]+"
            # Add regex expression after putting a comma to blacklist a disk
            # If value is empty, the default regex will be used.
            - name: DISCOVER_DAEMON_UDEV_BLACKLIST
              value: "(?i)dm-[0-9]+,(?i)rbd[0-9]+,(?i)nbd[0-9]+"

            # - name: DISCOVER_DAEMON_RESOURCES
            #   value: |
            #     resources:
            #       limits:
            #         cpu: 500m
            #         memory: 512Mi
            #       requests:
            #         cpu: 100m
            #         memory: 128Mi

            # Time to wait until the node controller will move Rook pods to other
            # nodes after detecting an unreachable node.
            # Pods affected by this setting are:
            # mgr, rbd, mds, rgw, nfs, PVC based mons and osds, and ceph toolbox
            # The value used in this variable replaces the default value of 300 secs
            # added automatically by k8s as Toleration for
            # <node.kubernetes.io/unreachable>
            # The total amount of time to reschedule Rook pods in healthy nodes
            # before detecting a <not ready node> condition will be the sum of:
            #  --> node-monitor-grace-period: 40 seconds (k8s kube-controller-manager flag)
            #  --> ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS: 5 seconds
            - name: ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS
              value: "5"

            # The name of the node to pass with the downward API
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # The pod name to pass with the downward API
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            # The pod namespace to pass with the downward API
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          # Recommended resource requests and limits, if desired
          #resources:
          #  limits:
          #    cpu: 500m
          #    memory: 512Mi
          #  requests:
          #    cpu: 100m
          #    memory: 128Mi

          #  Uncomment it to run lib bucket provisioner in multithreaded mode
          #- name: LIB_BUCKET_PROVISIONER_THREADS
          #  value: "5"

      # Uncomment it to run rook operator on the host network
      #hostNetwork: true
      volumes:
        - name: rook-config
          emptyDir: {}
        - name: default-config-dir
          emptyDir: {}
        - name: webhook-cert
          emptyDir: {}
# OLM: END OPERATOR DEPLOYMENT

my k8s cluster， I also set node labels and storage node stains to prevent other Pods from scheduling to this node, and only tolerable Pods are allowed to schedule to this node

kubectl get node -o wide
NAME    STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION           CONTAINER-RUNTIME
k8s1    Ready    control-plane   21h   v1.28.6   10.102.28.60   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.7.13
node1   Ready    storage-node    21h   v1.28.6   10.102.28.61   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.7.13
node2   Ready    storage-node    21h   v1.28.6   10.102.28.62   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.7.13
node3   Ready    storage-node    21h   v1.28.6   10.102.28.63   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.7.13
[root@k8s1 examples]

I also tried to create rgw within the cluster, because I set the taint, I have no way to deploy rook-ceph-rgw-my-store-a-b9b898c4d-hbmbd in the blemish In cluster.yaml and operator.yaml, I did not see signs of type CSI_CEPHFS_PLUGIN_TOLERATIONS tolerating rgw

kubectl  get pod -n rook-ceph
NAME                                              READY   STATUS      RESTARTS   AGE
csi-cephfsplugin-cmcgs                            2/2     Running     0          12m
csi-cephfsplugin-njv5k                            2/2     Running     0          12m
csi-cephfsplugin-provisioner-54b6c886c7-8p7qq     5/5     Running     0          12m
csi-cephfsplugin-provisioner-54b6c886c7-zhhz5     5/5     Running     0          12m
csi-cephfsplugin-vdv8v                            2/2     Running     0          12m
csi-rbdplugin-52gpp                               2/2     Running     0          12m
csi-rbdplugin-provisioner-5685d999c4-8v8dl        5/5     Running     0          12m
csi-rbdplugin-provisioner-5685d999c4-x67tn        5/5     Running     0          12m
csi-rbdplugin-q6npv                               2/2     Running     0          12m
csi-rbdplugin-xjlbg                               2/2     Running     0          12m
rook-ceph-crashcollector-node1-7c7594574b-dq2r7   1/1     Running     0          10m
rook-ceph-crashcollector-node2-6df895c49f-98rxw   1/1     Running     0          10m
rook-ceph-crashcollector-node3-55548c4d64-mjfss   1/1     Running     0          10m
rook-ceph-mgr-a-6757995cf-4cs4q                   3/3     Running     0          11m
rook-ceph-mgr-b-57b5c7c754-mq7x7                  3/3     Running     0          11m
rook-ceph-mon-a-6c88d7f8f6-zt7hp                  2/2     Running     0          12m
rook-ceph-mon-b-595dd7b8dd-h88nx                  2/2     Running     0          11m
rook-ceph-mon-c-78bb479599-v5rcd                  2/2     Running     0          11m
rook-ceph-operator-6fc6c6d985-ftsxg               1/1     Running     0          12m
rook-ceph-osd-0-7fbb58f877-gmqxn                  2/2     Running     0          10m
rook-ceph-osd-1-77976b8b55-766q2                  2/2     Running     0          10m
rook-ceph-osd-2-587f979889-7zrmz                  2/2     Running     0          10m
rook-ceph-osd-prepare-node1-sw7j4                 0/1     Completed   0          10m
rook-ceph-osd-prepare-node2-78qjg                 0/1     Completed   0          10m
rook-ceph-osd-prepare-node3-b898d                 0/1     Completed   0          10m
rook-ceph-rgw-my-store-a-b9b898c4d-hbmbd          0/2     Pending     0          8m52s

I also tried to define cr using tolerations, but this field was not implemented in the estimation code, and an error occurred

strict decoding error: unknown field "spec.tolerations"

yaml

apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: my-store
  namespace: rook-ceph
spec:
  tolerations:
  - key: "node-role.kubernetes.io/storage-node"
    operator: "Exists"
    effect: "NoSchedule"
  metadataPool:
    failureDomain: host
    replicated:
      size: 3
  dataPool:
    failureDomain: host
    erasureCoded:
      dataChunks: 2
      codingChunks: 1
  preservePoolsOnDelete: true
  gateway:
    sslCertificateRef:
    port: 80
    # securePort: 443
    instances: 1
  healthCheck:
    bucket:
      disabled: false
      interval: 60s

[ehsan310]

The thing about your config is , you have rook cluster that is running inside the cluster, if you want to use external ceph cluster you need to create a rook-cluster with external true then you can use that one to create RGW to connect external ceph cluster.

if you want to have 2 ceph cluster one inside and another cluster outside of your k8s , then you need to create antoher rook cluster with external flag true

[parth-gr]

You need to follow the full documentation, https://rook.io/docs/rook/latest-release/CRDs/Cluster/external-cluster/external-cluster/

2024-05-15 04:08:31.937757 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/external-store". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: Secret "rgw-admin-ops-user" not found

`) First run python script with --rgw-endpoint flag https://rook.io/docs/rook/latest-release/CRDs/Cluster/external-cluster/external-cluster/#1-create-all-users-and-keys
2) then run the import script to get the radosgw admin secret created.

If you have the secret created then you can create the external object-store.yaml

Plus you have to use the cluster-external.yaml

[kubecto]

您需要遵循完整文档，https://rook.io/docs/rook/latest-release/CRDs/Cluster/external-cluster/external-cluster/

2024-05-15 04:08:31.937757 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/external-store". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: Secret "rgw-admin-ops-user" not found

`）首先使用 --rgw-endpoint 标志运行 python 脚本https://rook.io/docs/rook/latest-release/CRDs/Cluster/external-cluster/external-cluster/#1-create-all-users- and-keys 2) 然后运行导入脚本以创建 radosgw 管理密钥。

如果您创建了秘密，那么您可以创建外部 object-store.yaml

另外，您必须使用cluster-external.yaml

Is it necessary to create another rook cluster to connect external ceph clusters as ehsan310 said? Why can't you use a rook cluster to connect multiple external clusters and just create an external rgw connection? Why can't rook be managed in a unified manner? A rook management itself can also manage multiple external clusters, but just create a few more CRS, which looks simpler to manage

[kubecto]

关于您的配置的事情是，您有在集群内部运行的 rook 集群，如果您想使用外部 ceph 集群，您需要创建一个具有外部的 rook 集群，true然后您可以使用该集群创建 RGW 来连接外部 ceph 集群。

如果你想有 2 个 ceph 集群，一个在 k8s 内部，另一个在 k8s 外部，那么你需要创建一个外部标志为 true 的 antoher rook 集群

关于您的配置的事情是，您有在集群内部运行的 rook 集群，如果您想使用外部 ceph 集群，您需要创建一个具有外部的 rook 集群，true然后您可以使用该集群配置 RGW 来连接外部 ceph 集群。

如果您想要有 2 个 ceph 集群，一个在 k8s 内部，另一个在 k8s 外部，那么您需要创建一个外部标志为 true 的 antoher rook 集群

What configuration should I refer to for interconnection? , there is no reference link.

[ehsan310]

您需要遵循完整文档，https://rook.io/docs/rook/latest-release/CRDs/Cluster/external-cluster/external-cluster/

2024-05-15 04:08:31.937757 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/external-store". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: Secret "rgw-admin-ops-user" not found

`）首先使用 --rgw-endpoint 标志运行 python 脚本https://rook.io/docs/rook/latest-release/CRDs/Cluster/external-cluster/external-cluster/#1-create-all-users- and-keys 2) 然后运行导入脚本以创建 radosgw 管理密钥。
如果您创建了秘密，那么您可以创建外部 object-store.yaml
另外，您必须使用cluster-external.yaml

Is it necessary to create another rook cluster to connect external ceph clusters as ehsan310 said? Why can't you use a rook cluster to connect multiple external clusters and just create an external rgw connection? Why can't rook be managed in a unified manner? A rook management itself can also manage multiple external clusters, but just create a few more CRS, which looks simpler to manage

you are use rook-oprator to create multiple rook-cluster.
rook oprtator is handling multiple cluster one can be internal and another external.

[kubecto]

ok, it seems that I still need to experiment a few more times. Currently, I am not familiar with rook. Besides, why does CephObjectStore not support tolerance

[parth-gr]

@kubecto use this document to connect rook to other rook clusters externally https://rook.io/docs/rook/latest-release/CRDs/Cluster/external-cluster/external-cluster/#exporting-rook-to-another-cluster

[BlaineEXE]

Please be advised that Rook v1.10 is unsupported. Rook only provides support for the most recent 2 versions, which are v1.13 and v1.14 currently. Especially if you are new to Rook and experimenting, I would highly recommend that you begin with v1.14.

[travisn]

From the original issue, it looks like you are also connecting to Ceph v14 nautilus, which has been out of support for some time. Are you able to upgrade Ceph soon, as well as to a newer version of Rook? See also Rook's official release cycle doc. We will still attempt to answer questions, but it is more difficult when such older versions are in use.

[kubecto]

OK, problem solved. I'll shut it down