How to Increase MAX_OSD Count Declarative

[petkovp]

I want to increase MAX_OSD count to 100, the default values=9

bash-5.1$ ceph osd getmaxosd
max_osd = 9 in epoch 947

ID   CLASS  WEIGHT   TYPE NAME                                STATUS  REWEIGHT  PRI-AFF
 -1         1.17178  root default
-15         0.19530      host pvc-s3-bucket-cold-data-0cs2hx
  7   cold  0.19530          osd.7                                up   1.00000  1.00000
-17         0.19530      host pvc-s3-bucket-cold-data-18r9l8
  6   cold  0.19530          osd.6                                up   1.00000  1.00000
-19         0.19530      host pvc-s3-bucket-cold-data-2wbpn9
  8   cold  0.19530          osd.8                                up   1.00000  1.00000
 -5         0.19530      host pvc-s3-bucket-data-0lp4v8
  1    hdd  0.19530          osd.1                                up   1.00000  1.00000
 -7         0.19530      host pvc-s3-bucket-data-17fbpl
  2    hdd  0.19530          osd.2                                up   1.00000  1.00000
 -3         0.19530      host pvc-s3-bucket-data-2xc65v
  0    hdd  0.19530          osd.0                                up   1.00000  1.00000

I am using helm chart in specified following key-value pair max_osd = 100 in config-map rook-config-override, but changes didn't applied and cluster doesn't access the new PVC and OSD.

Could you please share any information how I can increase the value.

[travisn]

max_osd does not limit how many OSDs can be created, I just expect it is reporting how many OSDs there are. There is no need to configure it.

[petkovp]

I have 6 osd (osd.0, osd.1, osd.2, osd.6, osd.7 and osd.8) I try to scale my cluster with one more osd, but osd doesn't created
The rook-ceph cluster configuration and events

Name:         rook-ceph
Namespace:    rook-ceph
Labels:       argocd.argoproj.io/instance=bgsd-rancher-dev_rook-ceph-cluster
Annotations:  argocd.argoproj.io/tracking-id: bgsd-rancher-dev_rook-ceph-cluster:ceph.rook.io/CephCluster:rook-ceph/rook-ceph
API Version:  ceph.rook.io/v1
Kind:         CephCluster
Metadata:
  Creation Timestamp:  2026-03-06T11:08:55Z
  Finalizers:
    cephcluster.ceph.rook.io
  Generation:        28
  Resource Version:  349119097
  UID:               4d853e57-f817-4e97-bffe-2ea3199120fc
Spec:
  Ceph Version:
    Allow Unsupported:  false
    Image:              quay.io/ceph/ceph:v19.2.3
  Cleanup Policy:
    Allow Uninstall With Volumes:  false
    Confirmation:
    Sanitize Disks:
      Data Source:                                    zero
      Iteration:                                      1
      Method:                                         quick
    Wipe Devices From Other Clusters:                 false
  Continue Upgrade After Checks Even If Not Healthy:  true
  Crash Collector:
    Disable:  false
  Csi:
    Cephfs:
    Read Affinity:
      Enabled:  false
  Dashboard:
    Enabled:                         true
    Prometheus Endpoint:             http://kube-prometheus-stack-prometheus.infrastructure.svc:9090
    Prometheus Endpoint SSL Verify:  false
    Ssl:                             false
  Data Dir Host Path:                /var/lib/rook
  Disruption Management:
    Manage Pod Budgets:       true
    Osd Maintenance Timeout:  30
  External:
  Health Check:
    Daemon Health:
      Mon:
        Disabled:  false
        Interval:  45s
      Osd:
        Disabled:  false
        Interval:  60s
      Status:
        Disabled:  false
        Interval:  60s
    Liveness Probe:
      Mgr:
        Disabled:  false
      Mon:
        Disabled:  false
      Osd:
        Disabled:  false
  Labels:
    Monitoring:
      Release:  kube-prometheus-stack
  Log Collector:
    Enabled:       true
    Max Log Size:  500M
    Periodicity:   daily
  Mgr:
    Allow Multiple Per Node:  false
    Count:                    2
  Mon:
    Allow Multiple Per Node:  false
    Count:                    3
  Monitoring:
    Enabled:  true
  Network:
    Connections:
      Compression:
        Enabled:  false
      Encryption:
        Enabled:     false
      requireMsgr2:  false
    Multi Cluster Service:
  Priority Class Names:
    Mgr:                                   system-cluster-critical
    Mon:                                   system-node-critical
    Osd:                                   system-node-critical
  Remove OS Ds If Out And Safe To Remove:  false
  Resources:
    Cleanup:
      Limits:
        Memory:  1Gi
      Requests:
        Cpu:     500m
        Memory:  100Mi
    Crashcollector:
      Limits:
        Memory:  60Mi
      Requests:
        Cpu:     100m
        Memory:  60Mi
    Exporter:
      Limits:
        Memory:  128Mi
      Requests:
        Cpu:     50m
        Memory:  50Mi
    Logcollector:
      Limits:
        Memory:  1Gi
      Requests:
        Cpu:     100m
        Memory:  100Mi
    Mgr:
      Limits:
        Memory:  1Gi
      Requests:
        Cpu:     500m
        Memory:  512Mi
    Mgr - Sidecar:
      Limits:
        Memory:  100Mi
      Requests:
        Cpu:     100m
        Memory:  40Mi
    Mon:
      Limits:
        Memory:  2Gi
      Requests:
        Cpu:     1000m
        Memory:  1Gi
    Osd:
      Limits:
        Memory:  4Gi
      Requests:
        Cpu:     1000m
        Memory:  4Gi
    Prepareosd:
      Requests:
        Cpu:     500m
        Memory:  50Mi
  Security:
    Cephx:
      Csi:
      Daemon:
      Rbd Mirror Peer:
    Key Rotation:
      Enabled:  false
    Kms:
  Skip Upgrade Checks:  false
  Storage:
    Allow Device Class Update:        true
    Allow Osd Crush Weight Update:    true
    Flapping Restart Interval Hours:  0
    Migration:
    Storage Class Device Sets:
      Count:     4
      Name:      pvc-s3-bucket
      Portable:  true
      Volume Claim Templates:
        Metadata:
          Name:  data
        Spec:
          Access Modes:
            ReadWriteOnce
          Resources:
            Requests:
              Storage:         200Gi
          Storage Class Name:  vmware-sc
          Volume Mode:         Block
      Config:
        Device Class:  cold
      Count:           3
      Name:            pvc-s3-bucket-cold
      Portable:        true
      Volume Claim Templates:
        Metadata:
          Annotations:
            Crush Device Class:  cold
          Name:                  data
        Spec:
          Access Modes:
            ReadWriteOnce
          Resources:
            Requests:
              Storage:         200Gi
          Storage Class Name:  vmware-sc
          Volume Mode:         Block
    Store:
    Use All Devices:                        false
    Use All Nodes:                          false
  Upgrade OSD Requires Healthy P Gs:        false
  Wait Timeout For Healthy OSD In Minutes:  10
Status:
  Ceph:
    Capacity:
      Bytes Available:  1284312489984
      Bytes Total:      1288490188800
      Bytes Used:       4177698816
      Last Updated:     2026-03-23T16:12:52Z
    Details:
      MON_DISK_LOW:
        Message:      mons b,c are low on available space
        Severity:     HEALTH_WARN
    Fsid:             9719f8c4-9b0c-4b35-929c-877db7058d4b
    Health:           HEALTH_WARN
    Last Changed:     2026-03-19T18:22:50Z
    Last Checked:     2026-03-23T16:12:52Z
    Previous Health:  HEALTH_ERR
    Versions:
      Mgr:
        ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable):  2
      Mon:
        ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable):  3
      Osd:
        ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable):  6
      Overall:
        ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable):  12
      Rgw:
        ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable):  1
  Cephx:
    Admin:
    Ceph Exporter:
      Key Ceph Version:  19.2.3-0
      Key Generation:    1
    Crash Collector:
      Key Ceph Version:  19.2.3-0
      Key Generation:    1
    Csi:
      Key Ceph Version:  19.2.3-0
      Key Generation:    1
    Mgr:
      Key Ceph Version:  19.2.3-0
      Key Generation:    1
    Mon:
      Key Ceph Version:  19.2.3-0
      Key Generation:    1
    Osd:
      Key Ceph Version:  19.2.3-0
      Key Generation:    1
    Rbd Mirror Peer:
      Key Ceph Version:  19.2.3-0
      Key Generation:    1
  Conditions:
    Last Heartbeat Time:   2026-03-20T13:59:56Z
    Last Transition Time:  2026-03-06T11:14:53Z
    Message:               Cluster created successfully
    Reason:                ClusterCreated
    Status:                True
    Type:                  Ready
    Last Heartbeat Time:   2026-03-23T16:12:51Z
    Last Transition Time:  2026-03-23T16:12:51Z
    Message:               Configuring Ceph Mons
    Reason:                ClusterProgressing
    Status:                True
    Type:                  Progressing
  Message:                 Configuring Ceph Mons
  Observed Generation:     24
  Phase:                   Progressing
  State:                   Creating
  Storage:
    Device Classes:
      Name:  hdd
      Name:  cold
    Osd:
      Migration Status:
      Store Type:
        Bluestore:  6
  Version:
    Image:    quay.io/ceph/ceph:v19.2.3
    Version:  19.2.3-0
Events:
  Type     Reason           Age                 From                          Message
  ----     ------           ----                ----                          -------
  Warning  ReconcileFailed  34m (x19 over 61m)  rook-ceph-cluster-controller  failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to execute post actions after all the ceph managers started: failed to set config store options: failed to set ceph config for target: osd: failed to set keys [max_osd]
  Warning  ReconcileFailed  22m (x7 over 23m)   rook-ceph-cluster-controller  failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to initialize ceph cluster info: failed to save mons: failed to write connection config for new mons: failed to write connection config: failed to write config to /var/lib/rook/rook-ceph: failed to merge global config with "rook-config-override": failed to load config data from "rook-config-override": key-value delimiter not found: max_osd 100
  Warning  ReconcileFailed  12s                 rook-ceph-cluster-controller  failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: failed to update/create OSDs: aborting OSD provisioning after waiting more than 20 minutes for OSDs to finish processing

POD: rook-ceph-osd-prepare-pvc-s3-bucket-data-3ddcc2-n62hn

[2026-03-23 16:16:29,940][ceph_volume.process][INFO  ] stderr 2026-03-23T16:16:29.667+0000 79e78145ea80 -1 bluestore(/mnt/pvc-s3-bucket-data-3ddcc2) No valid bdev label found
[2026-03-23 16:16:29,940][ceph_volume.process][INFO  ] stderr unable to read label for /mnt/pvc-s3-bucket-data-3ddcc2: (2) No such file or directory
[2026-03-23 16:16:29,940][ceph_volume.process][INFO  ] Running command: /usr/sbin/blkid -c /dev/null -p /mnt/pvc-s3-bucket-data-3ddcc2
[2026-03-23 16:16:29,949][ceph_volume.process][INFO  ] Running command: /usr/sbin/blkid -c /dev/null -p /mnt/pvc-s3-bucket-data-3ddcc2
[2026-03-23 16:16:29,956][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph-authtool --gen-print-key
[2026-03-23 16:16:29,984][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph-authtool --gen-print-key
[2026-03-23 16:16:30,012][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 3a743b2e-562c-4afa-8f75-05b638b5c5ff
[2026-03-23 16:16:30,451][ceph_volume.process][INFO  ] stderr Error EEXIST: entity osd.3 exists but key does not match
[2026-03-23 16:16:30,451][ceph_volume.objectstore.rawbluestore][ERROR ] raw prepare was unable to complete
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/ceph_volume/objectstore/rawbluestore.py", line 74, in safe_prepare
    self.prepare()
  File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.9/site-packages/ceph_volume/objectstore/rawbluestore.py", line 98, in prepare
    self.osd_id = prepare_utils.create_id(
  File "/usr/lib/python3.9/site-packages/ceph_volume/util/prepare.py", line 152, in create_id
    raise RuntimeError('Unable to create a new OSD id')
RuntimeError: Unable to create a new OSD id
[2026-03-23 16:16:30,452][ceph_volume.objectstore.rawbluestore][INFO  ] will rollback OSD ID creation
[2026-03-23 16:16:30,452][ceph_volume][ERROR ] exception caught by decorator
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 59, in newfunc
    return f(*a, **kw)
  File "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 166, in main
    terminal.dispatch(self.mapper, subcommand_args)
  File "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.9/site-packages/ceph_volume/devices/raw/main.py", line 32, in main
    terminal.dispatch(self.mapper, self.argv)
  File "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.9/site-packages/ceph_volume/devices/raw/prepare.py", line 54, in main
    self.objectstore.safe_prepare(self.args)
  File "/usr/lib/python3.9/site-packages/ceph_volume/objectstore/rawbluestore.py", line 74, in safe_prepare
    self.prepare()
  File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.9/site-packages/ceph_volume/objectstore/rawbluestore.py", line 98, in prepare
    self.osd_id = prepare_utils.create_id(
  File "/usr/lib/python3.9/site-packages/ceph_volume/util/prepare.py", line 152, in create_id
    raise RuntimeError('Unable to create a new OSD id')
RuntimeError: Unable to create a new OSD id
: exit status 1

[travisn]

There is an error when creating the new OSD because of the max_osd setting.

key-value delimiter not found: max_osd 100

Remove max_osd from your config, and hopefully then it can successfully create new OSDs.