Ceph containers aren't using specified dataDirHostPath

[Thingee]

Is this a bug report or feature request?

• Bug Report

Deviation from expected behavior:
Container data directory is still set to /var/lib/rook after setting the dataDirHostPath in ceph/cluster.yaml to /data/rook

Expected behavior:

For the container processes mon-data and osd-data to use /data/rook as specified in ceph/cluster.yaml

How to reproduce it (minimal and precise):

1. Edit rook/cluster/examples/kubernetes/ceph/cluster.yaml dataDirHostPath to something like /data/rook. This is due to issue Can't work on minikube #1192.
2. Run:

minikube start --vm-driver kvm2 --cpus 4 --memory 4096 --disk-size 1024g --mount-string /home/ceph:/minikube-host; cd ~/rook/cluster/examples/kubernetes; kubectl create -f ceph/operator.yaml; kubectl create -f ceph/cluster.yaml; kubectl create -f mysql.yaml; kubectl create -f wordpress.yaml

3. minikube ssh

$ ps fax | grep rook    
12268 pts/0    S+     0:00              \_ grep rook
 6410 ?        Ss     0:00      |   \_ /tini -- /usr/local/bin/rook ceph operator
 6430 ?        Sl     0:00      |       \_ /usr/local/bin/rook ceph operator
 6632 ?        Ss     0:00      |   \_ /tini -- /usr/local/bin/rook ceph agent
 6649 ?        Sl     0:00      |       \_ /usr/local/bin/rook ceph agent
 6705 ?        Ss     0:00      |   \_ /tini -- /usr/local/bin/rook discover --discover-interval 60m
 6739 ?        Sl     0:00      |       \_ /usr/local/bin/rook discover --discover-interval 60m
 7735 ?        Ssl    0:00      |   \_ ceph-mon --foreground --public-addr 10.109.147.209:6790 --name mon.a --mon-data /var/lib/rook/mon-a/data
 8250 ?        Ssl    0:00      |   \_ ceph-mon --foreground --public-addr 10.97.168.128:6790 --name mon.b --mon-data /var/lib/rook/mon-b/data
 8770 ?        Ssl    0:00      |   \_ ceph-mon --foreground --public-addr 10.103.30.6:6790 --name mon.c --mon-data /var/lib/rook/mon-c/data
10746 ?        Ssl    0:00          \_ ceph-osd --foreground --id 0 --conf /var/lib/rook/osd0/rook-ceph.config --osd-data /var/lib/rook/osd0 --keyring /var/lib/rook/osd0/keyring --cluster rook-ceph --osd-uuid 63250f01-3f15-4762-a152-c0ccb096ee2e --osd-journal=/var/lib/rook/osd0/journal
$ ls /var/lib/rook
ls: cannot access '/var/lib/rook': No such file or directory
$ ls /data/rook
mon-a  mon-b  mon-c  osd0  rook-ceph

Environment:

• OS (e.g. from /etc/os-release): Fedora 29
• Kernel (e.g. uname -a): 4.19.6-300.fc29.x86_64
• Cloud provider or hardware configuration: Intel NUC6i7KYK Mini PC NUC Kit, Core i7-6770HQ 2.6GHz
• Rook version (use rook version inside of a Rook Pod): 0.9
• Kubernetes version (use kubectl version): 1.10.0
• Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): minikube
• Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):

[root@minikube /]# ceph status
  cluster:
    id:     cba25bc0-2d7b-4424-9e20-54f47814b821
    health: HEALTH_ERR
            2 modules have failed
 
  services:
    mon: 3 daemons, quorum b,c,a
    mgr: a(active)
    osd: 1 osds: 1 up, 1 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   55 GiB used, 778 GiB / 832 GiB avail
    pgs:     
 
[root@minikube /]# ceph osd status
+----+---------------------------------+-------+-------+--------+---------+--------+---------+-----------+
| id |               host              |  used | avail | wr ops | wr data | rd ops | rd data |   state   |
+----+---------------------------------+-------+-------+--------+---------+--------+---------+-----------+
| 0  | rook-ceph-osd-0-d89686885-x6xxr | 54.8G |  777G |    0   |     0   |    0   |     0   | exists,up |
+----+---------------------------------+-------+-------+--------+---------+--------+---------+-----------+
[root@minikube /]# ceph df
GLOBAL:
    SIZE        AVAIL       RAW USED     %RAW USED 
    832 GiB     778 GiB       55 GiB          6.59 
POOLS:
    NAME     ID     USED     %USED     MAX AVAIL     OBJECTS 
[root@minikube /]# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR 

total_objects    0
total_used       55 GiB
total_avail      778 GiB
total_space      832 GiB

[Thingee]

Likely related ... trying the mysql example and doing a pod describe gives an unbound volume claim ... unbound-volume-claim.txt

[jbw976]

@Thingee from this output, it looks like the mons and OSD are using /data/rook like you asked?

$ ls /data/rook
mon-a  mon-b  mon-c  osd0  rook-ceph

Note that ceph containers get the dataDirHostPath bind mounted to /var/lib/rook inside the container always, so from the perspective of the container's process mount namespace, it will be using /var/lib/rook always. But that should be bind mounted to /data/rook on the host machine, resulting in the directories we see from the ls output above.

[jbw976]

I also see the following bind mount from docker inspect on one of the ceph mon containers:

            {
                "Type": "bind",
                "Source": "/data/rook",
                "Destination": "/var/lib/rook",
                "Mode": "rslave",
                "RW": true,
                "Propagation": "rslave"
            },