volume folder created with unexpected different user and folder

[brightzheng100]

Some volume folders created with an unexpected different user user, which is my current OS login user, which is NOT expected.

$ ls -al /storage/
total 24
drwxrwxrwx  6 root    root    4096 Jan 10 05:13 .
drwxr-xr-x 20 root    root    4096 Jan 10 04:38 ..
drwxrwxrwx  3 root    root    4096 Jan 10 04:45 pvc-0ea1f977-fe0c-40e8-88f3-311f685f52bf_xxx-kafka_data-0-kafka-kafka-0
drwxrwxrwx  3 root    root    4096 Jan 10 04:38 pvc-6c7303e7-30c1-411d-be91-7e3d3615a000_xxx-postgres_postgres-1
drwxrwxrwx  7 root    root    4096 Jan 10 04:39 pvc-86e2e71c-1bcf-4cda-892f-129469dd3cb9_xxx-cassandra_server-data-xxx-cassandra-default-sts-0
drwxrwxrwx  2 user user 4096 Jan 10 05:13 pvc-eff01513-6998-4763-bf8d-624cfb34e685_xxx-elasticsearch_elasticsearch-data-elasticsearch-es-default-0

I defined my custom config like this -- Note that I tried to tweak the setup script to change the owner explicitly to be root:root but didn't help:

kind: ConfigMap
apiVersion: v1
metadata:
  name: local-path-config
  namespace: local-path-storage
data:
  config.json: |-
    {
      "nodePathMap":[
        {
          "node":"DEFAULT_PATH_FOR_NON_LISTED_NODES",
          "paths":["/storage"]
        }
      ]
    }
  setup: |-
    #!/bin/sh
    set -eu
    echo mkdir -m 0777 -p "$VOL_DIR"
    mkdir -m 0777 -p "$VOL_DIR"
    echo chown root:root -R "$VOL_DIR"
    chown root:root -R "$VOL_DIR"
    echo chmod 0777 -R "$VOL_DIR"
    chmod 0777 -R "$VOL_DIR"
  teardown: |-
    #!/bin/sh
    set -eu
    rm -rf "$VOL_DIR"
  helperPod.yaml: |-
    apiVersion: v1
    kind: Pod
    metadata:
      name: helper-pod
    spec:
      priorityClassName: system-node-critical
      tolerations:
        - key: node.kubernetes.io/disk-pressure
          operator: Exists
          effect: NoSchedule
      containers:
      - name: helper-pod
        image: busybox
        imagePullPolicy: IfNotPresent

And the complete logs while creating the volume:

I0110 05:13:16.410902       1 controller.go:1337] provision "xxx-elasticsearch/elasticsearch-data-elasticsearch-es-default-0" class "local-path-data": started
time="2024-01-10T05:13:16Z" level=debug msg="config doesn't contain node itz-550004ghs4-9lf9, use DEFAULT_PATH_FOR_NON_LISTED_NODES instead"
time="2024-01-10T05:13:16Z" level=info msg="Creating volume pvc-eff01513-6998-4763-bf8d-624cfb34e685 at itz-550004ghs4-9lf9:/storage/pvc-eff01513-6998-4763-bf8d-624cfb34e685_xxx-elasticsearch_elasticsearch-data-elasticsearch-es-default-0"
time="2024-01-10T05:13:16Z" level=info msg="create the helper pod helper-pod-create-pvc-eff01513-6998-4763-bf8d-624cfb34e685 into local-path-storage"
I0110 05:13:16.417871       1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"xxx-elasticsearch", Name:"elasticsearch-data-elasticsearch-es-default-0", UID:"eff01513-6998-4763-bf8d-624cfb34e685", APIVersion:"v1", ResourceVersion:"28154", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "xxx-elasticsearch/elasticsearch-data-elasticsearch-es-default-0"
time="2024-01-10T05:13:20Z" level=info msg="Volume pvc-eff01513-6998-4763-bf8d-624cfb34e685 has been created on itz-550004ghs4-9lf9:/storage/pvc-eff01513-6998-4763-bf8d-624cfb34e685_xxx-elasticsearch_elasticsearch-data-elasticsearch-es-default-0"
time="2024-01-10T05:13:20Z" level=info msg="Start of helper-pod-create-pvc-eff01513-6998-4763-bf8d-624cfb34e685 logs"
time="2024-01-10T05:13:20Z" level=info msg="mkdir -m 0777 -p /storage/pvc-eff01513-6998-4763-bf8d-624cfb34e685_xxx-elasticsearch_elasticsearch-data-elasticsearch-es-default-0"
time="2024-01-10T05:13:20Z" level=info msg="chown root:root -R /storage/pvc-eff01513-6998-4763-bf8d-624cfb34e685_xxx-elasticsearch_elasticsearch-data-elasticsearch-es-default-0"
time="2024-01-10T05:13:20Z" level=info msg="chmod 0777 -R /storage/pvc-eff01513-6998-4763-bf8d-624cfb34e685_xxx-elasticsearch_elasticsearch-data-elasticsearch-es-default-0"
time="2024-01-10T05:13:20Z" level=info msg="End of helper-pod-create-pvc-eff01513-6998-4763-bf8d-624cfb34e685 logs"
I0110 05:13:20.458905       1 controller.go:1442] provision "xxx-elasticsearch/elasticsearch-data-elasticsearch-es-default-0" class "local-path-data": volume "pvc-eff01513-6998-4763-bf8d-624cfb34e685" provisioned
I0110 05:13:20.458905       1 controller.go:1455] provision "xxx-elasticsearch/elasticsearch-data-elasticsearch-es-default-0" class "local-path-data": succeeded
I0110 05:13:20.458917       1 volume_store.go:212] Trying to save persistentvolume "pvc-eff01513-6998-4763-bf8d-624cfb34e685"
I0110 05:13:20.462941       1 volume_store.go:219] persistentvolume "pvc-eff01513-6998-4763-bf8d-624cfb34e685" saved
I0110 05:13:20.462972       1 controller.go:1337] provision "xxx-elasticsearch/elasticsearch-data-elasticsearch-es-default-0" class "local-path-data": started
I0110 05:13:20.462986       1 controller.go:1346] provision "xxx-elasticsearch/elasticsearch-data-elasticsearch-es-default-0" class "local-path-data": persistentvolume "pvc-eff01513-6998-4763-bf8d-624cfb34e685" already exists, skipping
I0110 05:13:20.462996       1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"xxx-elasticsearch", Name:"elasticsearch-data-elasticsearch-es-default-0", UID:"eff01513-6998-4763-bf8d-624cfb34e685", APIVersion:"v1", ResourceVersion:"28154", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-eff01513-6998-4763-bf8d-624cfb34e685

[brightzheng100]

And one more issue/bug that I tried multiple times but the issue remains: even I've customized the local-path-config configmap, some folders were still created in the default folder, instead of the desired configured /storage:

$ ls -al /opt/local-path-provisioner/
total 28
drwxrwxrwx  7 root root 4096 Jan 10 04:39 .
drwxr-xr-x  4 root root 4096 Jan 10 04:38 ..
drwxrwxrwx  2 root root 4096 Jan 10 04:44 pvc-057537df-9002-4eff-a29d-ff8a52732157_xxx-clickhouse_xxx-clickhouse-log-volume-chi-clickhouse-local-0-0-0
drwxrwxrwx  4 root root 4096 Jan 10 04:44 pvc-67889c0e-a0fa-4229-9a19-941c9526812b_xxx-kafka_data-kafka-zookeeper-0

[brightzheng100]

By adding more command in the setup script, I found that the folder initially created was with the right user root:

  setup: |-
    #!/bin/sh
    set -eu
    echo mkdir -m 0777 -p "$VOL_DIR"
    mkdir -m 0777 -p "$VOL_DIR"
    echo chown root:root -R "$VOL_DIR"
    chown root:root -R "$VOL_DIR"
    echo chmod 0777 -R "$VOL_DIR"
    chmod 0777 -R "$VOL_DIR"
    ls -al "$VOL_DIR"
    ls -al "$VOL_DIR"/../

...
time="2024-01-10T05:53:59Z" level=info msg="Start of helper-pod-create-pvc-5aca4075-cbc8-4168-8c24-fca8dd03cc04 logs"
time="2024-01-10T05:53:59Z" level=info msg="mkdir -m 0777 -p /storage/pvc-5aca4075-cbc8-4168-8c24-fca8dd03cc04_xxx-elasticsearch_elasticsearch-data-elasticsearch-es-default-0"
time="2024-01-10T05:53:59Z" level=info msg="chown root:root -R /storage/pvc-5aca4075-cbc8-4168-8c24-fca8dd03cc04_xxx-elasticsearch_elasticsearch-data-elasticsearch-es-default-0"
time="2024-01-10T05:53:59Z" level=info msg="chmod 0777 -R /storage/pvc-5aca4075-cbc8-4168-8c24-fca8dd03cc04_xxx-elasticsearch_elasticsearch-data-elasticsearch-es-default-0"
time="2024-01-10T05:53:59Z" level=info msg="total 8"
time="2024-01-10T05:53:59Z" level=info msg="drwxrwxrwx    2 root     root          4096 Jan 10 05:53 ."
time="2024-01-10T05:53:59Z" level=info msg="drwxrwxrwx    6 root     root          4096 Jan 10 05:53 .."
time="2024-01-10T05:53:59Z" level=info msg="total 24"
time="2024-01-10T05:53:59Z" level=info msg="drwxrwxrwx    6 root     root          4096 Jan 10 05:53 ."
time="2024-01-10T05:53:59Z" level=info msg="dr-xr-xr-x    1 root     root          4096 Jan 10 05:53 .."
time="2024-01-10T05:53:59Z" level=info msg="drwxrwxrwx    3 root     root          4096 Jan 10 04:45 pvc-0ea1f977-fe0c-40e8-88f3-311f685f52bf_xxx-kafka_data-0-kafka-kafka-0"
time="2024-01-10T05:53:59Z" level=info msg="drwxrwxrwx    2 root     root          4096 Jan 10 05:53 pvc-5aca4075-cbc8-4168-8c24-fca8dd03cc04_xxx-elasticsearch_elasticsearch-data-elasticsearch-es-default-0"
time="2024-01-10T05:53:59Z" level=info msg="drwxrwxrwx    3 root     root          4096 Jan 10 04:38 pvc-6c7303e7-30c1-411d-be91-7e3d3615a000_xxx-postgres_postgres-1"
time="2024-01-10T05:53:59Z" level=info msg="drwxrwxrwx    7 root     root          4096 Jan 10 04:39 pvc-86e2e71c-1bcf-4cda-892f-129469dd3cb9_xxx-cassandra_server-data-xxx-cassandra-default-sts-0"
time="2024-01-10T05:53:59Z" level=info msg="End of helper-pod-create-pvc-5aca4075-cbc8-4168-8c24-fca8dd03cc04 logs"
...

Not sure why the folder's owner was changed from initial root to user eventually.
Will dig deeper into the code.

[brightzheng100]

it turns out that this statement is a false alarm:

volume folder created with unexpected different user

As ElasticSearch operator will try to chown in its init container which will eventually change the user -- to be honest, not sure whether that's a good practice but that's what I observe.

But it's still true that volume folders may be created in the default /opt/local-path-provisioner and/or the configured folder, e.g. /storage.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.