MariaDB Operator - How it works?

[nandexsp]

Hi All!

I'm not sure how mariaDBOperator works. After setting accounting to True, you must set ExternalDB or MariaDBOperator. MariaDBOperator does not create a container with the image that comes in the cluster values when we activate it. Whether it is just activated to fill in the slurmdbd data or if a mariadb container is created somewhere is unknown to me.

Thanks

[dstaroff]

Hello @nandexsp !
In order to use accounting without external DB, you have to make sure that:

• MariaDB operator is installed into the k8s cluster. You can use this as a hint.
• Soperator Helm release value for controllerManager.manager.env.isMariadbCrdInstalled is set to true. Hint.
• Accounting section in Slurm Cluster CR is filled out like so.

[nandexsp]

oh, I see, MariaDB needs to be installed by your own. in this case, The mariaDB image from values: mariaDB: "docker-registry1.mariadb.com/library/mariadb:11.4.3" is not used

[Uburro]

oh, I see, MariaDB needs to be installed by your own. in this case, The mariaDB image from values: mariaDB: "docker-registry1.mariadb.com/library/mariadb:11.4.3" is not used

It is not used if controllerManager.manager.env.isMariadbCrdInstalled=false and mariadb-operator is not installed, yes

[dstaroff]

Let's unwind it:

• mariaDB image from values is an image used for MariaDB pod
• The pod itself is a component of MariaDB CR
• MariaDB CR is created by Soperator
• In order to transform MariaDB CR into useful pods, MariaDB operator is used. If it's not installed, no DB instance will be created, and the image will be unused

[nandexsp]

I see, I was trying to make a way to install in the same pod as slurmdbd but makes sense to managed by mariadb. I will create outside and create the NAMESPACE-acct-db host link ;)

[nandexsp]

I have the mariadb-operator and the CRDs installed but the MariaDB pod is not launched, with the accounting: true and this part:
mariadbOperator:
enabled: true
protectedSecret: true
resources:
cpu: "1000m"
memory: "1Gi"
ephemeralStorage: "5Gi"
replicas: 1

accounting is trying to connect to NAMESPACE-acct-db host and is not working for me. I tried to create a mariadb pod with the table and the service to connect to the host but it doesn't work. Have you got another way to do it?

Not sure if mariadb-operator and mariadb-operator-crds helms need to be installed in the same namespace or in the mariadb-operator namespace.
For some reason is not working and mariadboperator pod is not launched by soperator. Soperator value is enabled
isMariadbCrdInstalled: "true"

[Uburro]

Soperator has to create mariadb cr in same namespace where slurmcluster cr.
Show me
k -n nameSpaceName get mariadbs.k8s.mariadb.com
And
k -n nameSpaceName get sts

[nandexsp]

I didnt install the soperator-crds by helm, and now if I try to install it ....

$ helm install -n slurm-ai soperator-crds ./soperator-crds/
Error: INSTALLATION FAILED: Unable to continue with install: CustomResourceDefinition "nodeconfigurators.slurm.nebius.ai" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "soperator-crds"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "slurm-ai"

as it doesn't work I install it manually:

1. mariadb operator crds
   I just install the mariadb-operator-crds by helm:

helm install -n slurm-ai mariadb-operator-crds mariadb-operator/mariadb-operator-crds

and it "works" but mariadb operator is not created

nandex$ kubectl -n slurm-ai get mariadbs.k8s.mariadb.com
NAME                      READY   STATUS   PRIMARY   UPDATES                    AGE
slurm-ai-acct-db                              ReplicasFirstPrimaryLast   21s
nandex$ kubectl -n slurm-ai get sts
NAME         READY   AGE
controller   1/1     28s
login        1/1     27s
worker       1/2     27s

Accounting log:

accounting slurmdbd: error: The database must be up when starting the MYSQL plugin.  Trying again in 5 seconds.
accounting slurmdbd: error: mysql_real_connect failed: 2005 Unknown server host 'slurm-ai-acct-db' (-2)
accounting slurmdbd: error: The database must be up when starting the MYSQL plugin.  Trying again in 5 seconds.

Check if is mandatory to run mariadb-operator in the same ns.

If I launched the mariadb-operator I get this error in the mariadb-operator:
2. $ helm install -n slurm-ai mariadb-operator mariadb-operator/mariadb-operator

{"level":"error","ts":1740496908.2617135,"msg":"Reconciler error","controller":"mariadb","controllerGroup":"k8s.mariadb.com","controllerKind":"MariaDB","MariaDB":{"name":"slurm-ai-acct-db","namespace":"slurm-ai"},"namespace":"slurm-ai","name":"slurm-ai-acct-db","reconcileID":"192199e0-d4c3-4f4c-8857-908cf369946f","error":"error reconciling Spec: 1 error occurred:\n\t* admission webhook \"vmariadb.kb.io\" denied the request: spec.storage: Invalid value: v1alpha1.Storage{Ephemeral:(*bool)(0xc00154ac99), Size:(*resource.Quantity)(nil), StorageClassName:\"\", ResizeInUseVolumes:(*bool)(0xc00154ac9a), WaitForVolumeResize:(*bool)(0xc00154ac9b), VolumeClaimTemplate:(*v1alpha1.VolumeClaimTemplate)(nil)}: Either storage size or volumeClaimTemplate must be provided\n\n","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224"}

Finally I just create a PVC and add into the mariadboperator section:

    storage:
       ephemeral: false
       resizeInUseVolumes: false
       size: 10Gi
       storageClassName: mariadb-storage
       volumeClaimTemplate:
         metadata:
           name: mariadb-pvc

       waitForVolumeResize: false

and the pod is running. Next step try to configure it test it because the actuall error in the pod acct-db is Permission denied, probably about roles, I will check it

2025-02-25 16:18:16 0 [Warning] Can't create test file '/var/lib/mysql/slurm-ai-acct-db-0.lower-test' (Errcode: 13 "Permission denied")
/usr/sbin/mariadbd: Can't change dir to '/var/lib/mysql/' (Errcode: 13 "Permission denied")
2025-02-25 16:18:16 0 [ERROR] Aborting

Just solve the problem with the podSecurityContext flag:

      podSecurityContext:
        runAsUser: 999  # UID of mysql user inside the container
        runAsGroup: 999
        fsGroup: 999  # Ensures MariaDB can write to the volume
        fsGroupChangePolicy: "OnRootMismatch"# fsGroup: 2000
        runAsNonRoot: true

and now is working properly.

scontrol reconfigure and cluster will be ready

[nandexsp]

I was working in a way to use
MariaDbLocal: true
option and install in the slurmctld-slurmdbd container with something like this in the entrypoint.sh:

#!/bin/bash

set -e  # Exit immediately if any command returns a non-zero error

# Fetch the namespace dynamically
NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
echo "Detected Kubernetes Namespace: $NAMESPACE"

echo "Bind-mount REST JWT key from K8S secret"
touch /var/spool/slurmdbd/jwt_hs256.key
mount --bind /mnt/rest-jwt-key/rest_jwt.key /var/spool/slurmdbd/jwt_hs256.key

echo "Bind-mount slurm configs from K8S config map"
for file in /mnt/slurm-configs/*; do
    filename=$(basename "$file")
    touch "/etc/slurm/$filename" && mount --bind "$file" "/etc/slurm/$filename"
done

echo "Bind-mount slurm configs with secrets from K8S secrets"
for file in /mnt/slurm-secrets/*; do
    filename=$(basename "$file")
    touch "/etc/slurm/$filename" && mount --bind "$file" "/etc/slurm/$filename"
done

# Extract MariaDB password from SlurmDBD secret
MARIADB_PASSWORD=$(grep "^StoragePass=" /mnt/slurm-secrets/slurmdbd.conf | cut -d'=' -f2)

if [ -z "$MARIADB_PASSWORD" ]; then
    echo "ERROR: Failed to retrieve MariaDB password from slurmdbd.conf"
    exit 1
fi

echo "Updating /etc/hosts for MariaDB"
echo "127.0.0.1 ${NAMESPACE}-acct-db" >> /etc/hosts

echo "Enabling MariaDB service"
systemctl enable mariadb


echo "Configuring MariaDB settings in 50-server.cnf"
MARIADB_CONF="/etc/mysql/mariadb.conf.d/50-server.cnf"

# Replace only the first occurrence of `bind-address` and insert multiple lines
sed -i '0,/^bind-address[[:space:]]*=.*/s//bind-address=*\n\
default_storage_engine=InnoDB\n\
innodb_default_row_format=DYNAMIC\n\
innodb_buffer_pool_size=32768M\n\
innodb_log_file_size=64M\n\
innodb_lock_wait_timeout=900\n\
max_allowed_packet=16M/' "$MARIADB_CONF"

echo "Starting MariaDB service"
service mariadb start

echo "Creating Slurm database and user"
mysql -u root -e "
CREATE DATABASE IF NOT EXISTS slurm_acct_db;
CREATE USER IF NOT EXISTS 'slurm'@'localhost' IDENTIFIED BY '${MARIADB_PASSWORD}';
GRANT ALL PRIVILEGES ON slurm_acct_db.* TO 'slurm'@'localhost';
FLUSH PRIVILEGES;
"

echo "MariaDB setup completed successfully"

echo "Set permissions for shared /var/spool/slurmdbd"
chmod 755 /var/spool/slurmdbd # It changes permissions of this shared directory in other containers as well

# TODO: Since 1.29 kubernetes supports native sidecar containers. We can remove it in feature releases
echo "Waiting until munge started"
while [ ! -S "/run/munge/munge.socket.2" ]; do sleep 2; done

echo "Start slurmdbd daemon"
exec /usr/sbin/slurmdbd -D 2>&1 | tee >(multilog s100000000 n5 /var/log/slurm/multilog)

If you think that could be interesting I can add it into a PR.
and adding the mariadb-server into the dockerfile

[Uburro]

could you show logs soperator?

[Uburro]

    storage:
       ephemeral: false
       resizeInUseVolumes: false
       size: 10Gi
       storageClassName: mariadb-storage
       waitForVolumeResize: false

iIthink it's enough.
What kind of csi do you use?

[nandexsp]

Im using a nfs provisioner and is working, but I cannot allow to add replicas to the mariadbOperator system:

      replication:
        enabled: true
        primary:
          automaticFailover: true
          podIndex: 0
        probesEnabled: true
        replica:
          connectionRetries: 10
          connectionTimeout: "30s"
          gtid: CurrentPos 
          replPasswordSecretKeyRef:
            name: mariadb
            key: repl-password 
            generate: true
          syncTimeout: "60s" 
          waitPoint: AfterCommit  
        syncBinlog: true

but it doesn't work

any idea¿?

[nandexsp]

could you show logs soperator?

soperator

manager {"level":"info","ts":"2025-02-25T17:47:33Z","msg":"Reconciled common resources","controller":"slurmcluster","controllerGroup":"slurm.nebius.ai","controllerKind":"SlurmCluster","SlurmCluster":{"name":"slurm-ai","namespace":"slurm-ai"},"namespace":"slurm-ai","name":"slurm-ai","reconcileID":"a9556279-9689-45bf-b6a6-da1f44888c21"}
manager {"level":"info","ts":"2025-02-25T17:47:33Z","msg":"Reconciled Slurm Accounting","controller":"slurmcluster","controllerGroup":"slurm.nebius.ai","controllerKind":"SlurmCluster","SlurmCluster":{"name":"slurm-ai","namespace":"slurm-ai"},"namespace":"slurm-ai","name":"slurm-ai","reconcileID":"a9556279-9689-45bf-b6a6-da1f44888c21"}
manager {"level":"info","ts":"2025-02-25T17:47:33Z","msg":"Reconciled NCCL benchmark","controller":"slurmcluster","controllerGroup":"slurm.nebius.ai","controllerKind":"SlurmCluster","SlurmCluster":{"name":"slurm-ai","namespace":"slurm-ai"},"namespace":"slurm-ai","name":"slurm-ai","reconcileID":"a9556279-9689-45bf-b6a6-da1f44888c21"}
manager {"level":"info","ts":"2025-02-25T17:47:33Z","msg":"Reconciled Slurm Controllers","controller":"slurmcluster","controllerGroup":"slurm.nebius.ai","controllerKind":"SlurmCluster","SlurmCluster":{"name":"slurm-ai","namespace":"slurm-ai"},"namespace":"slurm-ai","name":"slurm-ai","reconcileID":"a9556279-9689-45bf-b6a6-da1f44888c21"}
manager {"level":"info","ts":"2025-02-25T17:47:33Z","msg":"Reconciled Slurm Workers","controller":"slurmcluster","controllerGroup":"slurm.nebius.ai","controllerKind":"SlurmCluster","SlurmCluster":{"name":"slurm-ai","namespace":"slurm-ai"},"namespace":"slurm-ai","name":"slurm-ai","reconcileID":"a9556279-9689-45bf-b6a6-da1f44888c21"}
manager {"level":"info","ts":"2025-02-25T17:47:33Z","msg":"Finished reconciliation of Slurm Cluster","controller":"slurmcluster","controllerGroup":"slurm.nebius.ai","controllerKind":"SlurmCluster","SlurmCluster":{"name":"slurm-ai","namespace":"slurm-ai"},"namespace":"slurm-ai","name":"slurm-ai","reconcileID":"a9556279-9689-45bf-b6a6-da1f44888c21"}
manager {"level":"info","ts":"2025-02-25T17:47:34Z","msg":"Starting reconciliation of Slurm Cluster","controller":"slurmcluster","controllerGroup":"slurm.nebius.ai","controllerKind":"SlurmCluster","SlurmCluster":{"name":"slurm-ai","namespace":"slurm-ai"},"namespace":"slurm-ai","name":"slurm-ai","reconcileID":"6341aa30-60bc-4187-8475-0013f042e56a"}
manager {"level":"info","ts":"2025-02-25T17:47:34Z","msg":"Reconciled Populate jail Job","controller":"slurmcluster","controllerGroup":"slurm.nebius.ai","controllerKind":"SlurmCluster","SlurmCluster":{"name":"slurm-ai"

Accounting:

accounting slurmdbd: accounting_storage/as_mysql: _check_mysql_concat_is_sane: MySQL server version is: 11.4.3-MariaDB-ubu2404
accounting slurmdbd: accounting_storage/as_mysql: init: Accounting storage MYSQL plugin loaded
accounting slurmdbd: slurmdbd version 24.05.5 started

slurm-ai-acct-db-0:

2025-02-25 17:52:44 0 [Note] InnoDB: Compressed tables use zlib 1.3
2025-02-25 17:52:44 0 [Note] InnoDB: Number of transaction pools: 1
2025-02-25 17:52:44 0 [Note] InnoDB: Using AVX512 instructions
2025-02-25 17:52:44 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2025-02-25 17:52:44 0 [Note] InnoDB: Initializing buffer pool, total size = 32.000GiB, chunk size = 512.000MiB
2025-02-25 17:52:44 0 [Note] InnoDB: Completed initialization of buffer pool
2025-02-25 17:52:44 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
2025-02-25 17:52:44 0 [Note] InnoDB: End of log at LSN=397121
2025-02-25 17:52:44 0 [Note] InnoDB: Opened 3 undo tablespaces
2025-02-25 17:52:44 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
2025-02-25 17:52:44 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2025-02-25 17:52:44 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2025-02-25 17:52:44 0 [Note] InnoDB: log sequence number 397121; transaction id 363
  2025-02-25 17:52:44 0 [Note] Plugin 'FEEDBACK' is disabled.
  2025-02-25 17:52:44 0 [Note] Plugin 'wsrep-provider' is disabled.
2025-02-25 17:52:44 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2025-02-25 17:52:44 0 [Note] Server socket created on IP: '0.0.0.0'.
2025-02-25 17:52:44 0 [Note] Server socket created on IP: '::'.
2025-02-25 17:52:44 0 [Note] mariadbd: Event Scheduler: Loaded 0 events
2025-02-25 17:52:44 0 [Note] InnoDB: Buffer pool(s) load completed at 250225 17:52:44
2025-02-25 17:52:44 0 [Note] mariadbd: ready for connections.
Version: '11.4.3-MariaDB-ubu2404'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution