should upgrade rook to 1.0 with rook-storage-service docs to make it work with the current develop branch

[hoatle]

to the latest version (1.0) and make sure everything works.

Currently, I could not use the storageclass for a pod to run

  Warning  FailedMount       6s (x4 over 6m55s)  kubelet, k8s-01    Unable to mount volumes for pod "nginx-deployment-5dd94c99d-n7tpd_nginx-example(135d0da5-2193-11e9-8b79-08002702ccc4)": timeout expired waiting for volumes to attach or mount for pod "nginx-example"/"nginx-deployment-5dd94c99d-n7tpd". list of unmounted volumes=[nginx-logs]. list of unattached volumes=[nginx-logs default-token-s789z]

it works with teracy-dev-k8s v0.3.0, but not with the current develop branch.

[hoatle]

rook 0.9 works on teracy-dev v0.3.0, rook 1.0 did not work on teracy-dev v0.3.0 with my check, the difference was that:

rook 0.9 created OSD when preparing, but rook 1.0 did not.

• rook 0.9:

2019-05-05 15:05:39.479216 I | cephcmd: desired devices to configure osds: [{Name: OSDsPerDevice:1 IsFilter:false}]
2019-05-05 15:05:39.481089 I | rookcmd: starting Rook v0.9.3 with arguments '/rook/rook ceph osd provision'
2019-05-05 15:05:39.481115 I | rookcmd: flag values: --cluster-id=b29a0a52-6f46-11e9-96d8-080027c2be11, --data-device-filter=, --data-devices=, --data-directories=, --encrypted-device=false, --force-format=false, --help=false, --location=, --log-level=INFO, --metadata-device=, --node-name=k8s-01, --osd-database-size=1024, --osd-journal-size=1024, --osd-store=, --osd-wal-size=576, --osds-per-device=1
2019-05-05 15:05:39.481120 I | cephmon: parsing mon endpoints: a=10.233.51.254:6790,b=10.233.7.228:6790,c=10.233.40.1:6790
2019-05-05 15:05:39.502188 W | cephconfig: failed to add config file override from '/etc/rook/config/override.conf': open /etc/rook/config/override.conf: no such file or directory
2019-05-05 15:05:39.502211 I | cephconfig: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2019-05-05 15:05:39.502276 I | cephconfig: copying config to /etc/ceph/ceph.conf
2019-05-05 15:05:39.502348 I | cephconfig: generated admin config in /var/lib/rook/rook-ceph
2019-05-05 15:05:39.502355 I | cephosd: discovering hardware
2019-05-05 15:05:39.502363 I | exec: Running command: lsblk --all --noheadings --list --output KNAME
2019-05-05 15:05:39.512706 I | exec: Running command: lsblk /dev/loop1 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 15:05:39.515359 W | inventory: skipping device loop1: Failed to complete 'lsblk /dev/loop1': exit status 1. lsblk: /dev/loop1: not a block device
2019-05-05 15:05:39.515388 I | exec: Running command: lsblk /dev/loop6 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 15:05:39.518046 W | inventory: skipping device loop6: Failed to complete 'lsblk /dev/loop6': exit status 1. lsblk: /dev/loop6: not a block device
2019-05-05 15:05:39.518074 I | exec: Running command: lsblk /dev/loop4 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 15:05:39.520983 W | inventory: skipping device loop4: Failed to complete 'lsblk /dev/loop4': exit status 1. lsblk: /dev/loop4: not a block device
2019-05-05 15:05:39.521009 I | exec: Running command: lsblk /dev/loop2 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 15:05:39.523600 W | inventory: skipping device loop2: Failed to complete 'lsblk /dev/loop2': exit status 1. lsblk: /dev/loop2: not a block device
2019-05-05 15:05:39.523640 I | exec: Running command: lsblk /dev/loop0 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 15:05:39.526007 W | inventory: skipping device loop0: Failed to complete 'lsblk /dev/loop0': exit status 1. lsblk: /dev/loop0: not a block device
2019-05-05 15:05:39.526035 I | exec: Running command: lsblk /dev/sda --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 15:05:39.528242 W | inventory: skipping device sda: Failed to complete 'lsblk /dev/sda': exit status 1. lsblk: /dev/sda: not a block device
2019-05-05 15:05:39.528270 I | exec: Running command: lsblk /dev/sda1 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 15:05:39.533253 W | inventory: skipping device sda1: Failed to complete 'lsblk /dev/sda1': exit status 1. lsblk: /dev/sda1: not a block device
2019-05-05 15:05:39.533282 I | exec: Running command: lsblk /dev/loop7 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 15:05:39.535955 W | inventory: skipping device loop7: Failed to complete 'lsblk /dev/loop7': exit status 1. lsblk: /dev/loop7: not a block device
2019-05-05 15:05:39.536001 I | exec: Running command: lsblk /dev/loop5 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 15:05:39.540011 W | inventory: skipping device loop5: Failed to complete 'lsblk /dev/loop5': exit status 1. lsblk: /dev/loop5: not a block device
2019-05-05 15:05:39.540055 I | exec: Running command: lsblk /dev/loop3 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 15:05:39.542668 W | inventory: skipping device loop3: Failed to complete 'lsblk /dev/loop3': exit status 1. lsblk: /dev/loop3: not a block device
2019-05-05 15:05:39.542700 I | cephosd: creating and starting the osds
2019-05-05 15:05:39.551477 I | cephosd: configuring osd devices: {"Entries":{}}
2019-05-05 15:05:39.552602 I | exec: Running command: ceph-volume lvm batch --prepare
2019-05-05 15:05:39.675733 I | cephosd: no more devices to configure
2019-05-05 15:05:39.675770 I | exec: Running command: ceph-volume lvm list --format json
2019-05-05 15:05:40.439748 I | cephosd: 0 ceph-volume osd devices configured on this node
2019-05-05 15:05:40.439781 I | cephosd: devices = []
2019-05-05 15:05:40.444095 I | cephosd: configuring osd dirs: map[/var/lib/rook:-1]
2019-05-05 15:05:40.444618 I | exec: Running command: ceph osd create c62a35e2-41bd-4418-8d00-62f5a2d9b950 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/532137911
2019-05-05 15:05:41.202724 I | cephosd: successfully created OSD c62a35e2-41bd-4418-8d00-62f5a2d9b950 with ID 0
2019-05-05 15:05:41.202765 I | cephosd: osd.0 appears to be new, cleaning the root dir at /var/lib/rook/osd0
2019-05-05 15:05:41.203213 W | cephconfig: failed to add config file override from '/etc/rook/config/override.conf': open /etc/rook/config/override.conf: no such file or directory
2019-05-05 15:05:41.203236 I | cephconfig: writing config file /var/lib/rook/osd0/rook-ceph.config
2019-05-05 15:05:41.203451 I | cephconfig: copying config to /etc/ceph/ceph.conf
2019-05-05 15:05:41.203719 I | exec: Running command: ceph auth get-or-create osd.0 -o /var/lib/rook/osd0/keyring osd allow * mon allow profile osd --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format plain
2019-05-05 15:05:41.565690 I | cephosd: Initializing OSD 0 file system at /var/lib/rook/osd0...
2019-05-05 15:05:41.565857 I | exec: Running command: ceph mon getmap --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/469790122
...

• rook 1.0:

$ kubectl -n rook-ceph logs -f rook-ceph-osd-prepare-k8s-01-sdwnc provision 
2019-05-05 17:01:22.233295 I | cephcmd: desired devices to configure osds: [{Name: OSDsPerDevice:1 IsFilter:false}]
2019-05-05 17:01:22.236805 I | rookcmd: starting Rook v1.0.0 with arguments '/rook/rook ceph osd provision'
2019-05-05 17:01:22.236840 I | rookcmd: flag values: --cluster-id=11f599f5-6f57-11e9-96d8-080027c2be11, --data-device-filter=, --data-devices=, --data-directories=, --encrypted-device=false, --force-format=false, --help=false, --location=, --log-flush-frequency=5s, --log-level=INFO, --metadata-device=, --node-name=k8s-01, --osd-database-size=20480, --osd-journal-size=5120, --osd-store=, --osd-wal-size=576, --osds-per-device=1
2019-05-05 17:01:22.236849 I | op-mon: parsing mon endpoints: a=10.233.51.254:6789,b=10.233.7.228:6789,c=10.233.40.1:6789
2019-05-05 17:01:22.262744 W | cephconfig: failed to add config file override from '/etc/rook/config/override.conf': open /etc/rook/config/override.conf: no such file or directory
2019-05-05 17:01:22.262772 I | cephconfig: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2019-05-05 17:01:22.264451 I | cephconfig: copying config to /etc/ceph/ceph.conf
2019-05-05 17:01:22.264892 I | cephconfig: generated admin config in /var/lib/rook/rook-ceph
2019-05-05 17:01:22.264909 I | cephosd: discovering hardware
2019-05-05 17:01:22.264921 I | exec: Running command: lsblk --all --noheadings --list --output KNAME
2019-05-05 17:01:22.277383 I | exec: Running command: lsblk /dev/loop1 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 17:01:22.282384 W | inventory: skipping device loop1: Failed to complete 'lsblk /dev/loop1': exit status 1. lsblk: /dev/loop1: not a block device
2019-05-05 17:01:22.282687 I | exec: Running command: lsblk /dev/loop6 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 17:01:22.285645 W | inventory: skipping device loop6: Failed to complete 'lsblk /dev/loop6': exit status 1. lsblk: /dev/loop6: not a block device
2019-05-05 17:01:22.285774 I | exec: Running command: lsblk /dev/loop4 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 17:01:22.288634 W | inventory: skipping device loop4: Failed to complete 'lsblk /dev/loop4': exit status 1. lsblk: /dev/loop4: not a block device
2019-05-05 17:01:22.288719 I | exec: Running command: lsblk /dev/loop2 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 17:01:22.291515 W | inventory: skipping device loop2: Failed to complete 'lsblk /dev/loop2': exit status 1. lsblk: /dev/loop2: not a block device
2019-05-05 17:01:22.291602 I | exec: Running command: lsblk /dev/loop0 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 17:01:22.293592 W | inventory: skipping device loop0: Failed to complete 'lsblk /dev/loop0': exit status 1. lsblk: /dev/loop0: not a block device
2019-05-05 17:01:22.293672 I | exec: Running command: lsblk /dev/sda --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 17:01:22.296284 W | inventory: skipping device sda: Failed to complete 'lsblk /dev/sda': exit status 1. lsblk: /dev/sda: not a block device
2019-05-05 17:01:22.296460 I | exec: Running command: lsblk /dev/sda1 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 17:01:22.298711 W | inventory: skipping device sda1: Failed to complete 'lsblk /dev/sda1': exit status 1. lsblk: /dev/sda1: not a block device
2019-05-05 17:01:22.298799 I | exec: Running command: lsblk /dev/loop7 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 17:01:22.302456 W | inventory: skipping device loop7: Failed to complete 'lsblk /dev/loop7': exit status 1. lsblk: /dev/loop7: not a block device
2019-05-05 17:01:22.302556 I | exec: Running command: lsblk /dev/loop5 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 17:01:22.305133 W | inventory: skipping device loop5: Failed to complete 'lsblk /dev/loop5': exit status 1. lsblk: /dev/loop5: not a block device
2019-05-05 17:01:22.305234 I | exec: Running command: lsblk /dev/loop3 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2019-05-05 17:01:22.307893 W | inventory: skipping device loop3: Failed to complete 'lsblk /dev/loop3': exit status 1. lsblk: /dev/loop3: not a block device
2019-05-05 17:01:22.307983 I | cephosd: creating and starting the osds
2019-05-05 17:01:22.319217 I | cephosd: configuring osd devices: {"Entries":{}}
2019-05-05 17:01:22.319523 I | exec: Running command: ceph-volume lvm batch --prepare
2019-05-05 17:01:22.446669 I | cephosd: no more devices to configure
2019-05-05 17:01:22.448061 I | exec: Running command: ceph-volume lvm list --format json
2019-05-05 17:01:23.223184 I | cephosd: 0 ceph-volume osd devices configured on this node
2019-05-05 17:01:23.223211 I | cephosd: devices = []
2019-05-05 17:01:23.226530 I | cephosd: configuring osd dirs: map[]
2019-05-05 17:01:23.226701 I | cephosd: removing osd devices: {"metadata":null,"entries":[]}
2019-05-05 17:01:23.226714 I | cephosd: removing osd dirs: map[]
2019-05-05 17:01:23.226717 I | cephosd: saving osd dir map
2019-05-05 17:01:23.226724 I | cephosd: device osds:[]
dir osds: []

[hoatle]

use this cluster.yaml config file with rook 1.0 and teracy-dev-k8s v0.3.0 should work:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    # For the latest ceph images, see https://hub.docker.com/r/ceph/ceph/tags
    image: ceph/ceph:v14.2.1-20190430
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false
  dashboard:
    enabled: true
  storage:
    useAllNodes: true
    useAllDevices: true
    directories:
    - path: /var/lib/rook

note the directories config above.

[hoatle]

should be fixed by #85

[hoatle]

@hieptranquoc please verify that rook 1.0 works well on the current develop branch, following this docs https://github.com/teracyhq-incubator/teracy-dev-k8s/blob/develop/docs/rook-storage-service.md and you can deploy sample pods with PVC successfully usng the rook storageclass.

[hieptranquoc]

teracy-k8s on develop branch works well on the rook version 1.0!