New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Have to re-create operator to make OSD pods created on new node #2134

Closed
fkpwolf opened this Issue Sep 18, 2018 · 3 comments

Comments

Projects
None yet
2 participants
@fkpwolf
Copy link

fkpwolf commented Sep 18, 2018

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior:

I have to delete operator pods as Common Issues page suggested.

Expected behavior:

prepare and osd pod should be created automatically.

How to reproduce it (minimal and precise):

I create cluster with useAllDevices: true. Then add a new node to k8s cluster. rook-ceph-agent and rook-discover pods are created without error in pod log. And rook-discover will find all disk:

2018-09-18 07:28:22.176571 I | rookcmd: starting Rook v0.8.0-186.g3bcf882 with arguments '/usr/local/bin/rook discover'
2018-09-18 07:28:22.180247 I | rookcmd: flag values: --help=false, --log-level=INFO
2018-09-18 07:28:22.191953 I | rook-discover: updating device configmap
2018-09-18 07:28:22.192251 I | exec: Running command: lsblk --all --noheadings --list --output KNAME
2018-09-18 07:28:22.205825 I | exec: Running command: lsblk /dev/vdb --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2018-09-18 07:28:22.213307 I | exec: Running command: sgdisk --print /dev/vdb
2018-09-18 07:28:22.231767 I | exec: Running command: udevadm info --query=property /dev/vdb
2018-09-18 07:28:22.240699 I | exec: Running command: lsblk /dev/sda --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2018-09-18 07:28:22.243633 I | exec: Running command: sgdisk --print /dev/sda
2018-09-18 07:28:22.259051 I | exec: Running command: udevadm info --query=property /dev/sda
2018-09-18 07:28:22.261724 I | exec: Running command: lsblk /dev/vda --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2018-09-18 07:28:22.268111 I | exec: Running command: sgdisk --print /dev/vda
2018-09-18 07:28:22.282075 I | exec: Running command: udevadm info --query=property /dev/vda
2018-09-18 07:28:22.293361 I | exec: Running command: lsblk /dev/vda1 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
2018-09-18 07:28:22.301779 I | exec: Running command: udevadm info --query=property /dev/vda1
2018-09-18 07:28:22.304801 I | exec: Running command: lsblk /dev/vdb --bytes --pairs --output NAME,SIZE,TYPE,PKNAME
2018-09-18 07:28:22.307996 I | exec: Running command: udevadm info --query=property /dev/vdb
2018-09-18 07:28:22.310197 I | exec: Running command: lsblk /dev/sda --bytes --pairs --output NAME,SIZE,TYPE,PKNAME
2018-09-18 07:28:22.312617 I | exec: Running command: udevadm info --query=property /dev/sda
2018-09-18 07:28:22.314814 I | exec: Running command: lsblk /dev/vda --bytes --pairs --output NAME,SIZE,TYPE,PKNAME
2018-09-18 07:28:22.317369 I | exec: Running command: udevadm info --query=property /dev/vda1
2018-09-18 07:28:22.319617 I | exec: Running command: udevadm info --query=property /dev/vda
2018-09-18 07:28:22.321852 I | rook-discover: available devices: [{Name:vdb Parent: HasChildren:false DevLinks:/dev/disk/by-path/pci-0000:00:09.0 /dev/disk/by-path/virtio-pci-0000:00:09.0 Size:32212254720 UUID:34b4bc22-65ca-4de0-8661-dbbeef637f66 Serial: Type:disk Rotational:true Readonly:false Partitions:[] Filesystem: Vendor: Model: WWN: WWNVendorExtension: Empty:true} {Name:sda Parent: HasChildren:false DevLinks:/dev/disk/by-id/ata-QEMU_HARDDISK_QM00001 /dev/disk/by-label/cidata /dev/disk/by-path/pci-0000:00:01.1-ata-1 /dev/disk/by-uuid/2018-02-11-12-51-27-00 Size:374784 UUID:306779ce-3352-41bc-a8ec-b5112e93c9da Serial:QEMU_HARDDISK_QM00001 Type:disk Rotational:true Readonly:false Partitions:[] Filesystem:iso9660 Vendor: Model:QEMU_HARDDISK WWN: WWNVendorExtension: Empty:false} {Name:vda Parent: HasChildren:false DevLinks:/dev/disk/by-path/pci-0000:00:08.0 /dev/disk/by-path/virtio-pci-0000:00:08.0 Size:4294967296 UUID:5ccf644a-502b-427e-a90d-fda5f5de71ba Serial: Type:disk Rotational:true Readonly:false Partitions:[{Name:vda1 Size:4293918720 Label: Filesystem:ext4}] Filesystem: Vendor: Model: WWN: WWNVendorExtension: Empty:false}]
2018-09-18 07:28:52.343604 I | rook-discover: updating device configmap

/dev/vda is OS disk. /dev/vdb is prepared for OSD. But I didn't find prepare and osd pod. After re-create operator pod, OSD pods was created as excepted.

Environment:

  • OS (e.g. from /etc/os-release): Fedora 28 (Cloud Edition)
  • Kernel (e.g. uname -a): 4.16.3-301.fc28.x86_64
  • Rook version (use rook version inside of a Rook Pod): v0.8.0-186.g3bcf882
  • Kubernetes version (use kubectl version): v1.11.3
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): kubeadm

@fkpwolf fkpwolf changed the title Have to re-create operator to create OSD pods on new node Have to re-create operator to make OSD pods created on new node Sep 18, 2018

@fkpwolf

This comment has been minimized.

Copy link

fkpwolf commented Sep 27, 2018

Just checked code and found rook only watch Cluster CRD currently. This CRD is rook's resource and doesn't related with k8s cluster. So when new node joined k8s cluster, the Ceph Cluster CRD will be treated as unmodified. But when I set useAllDevices: ture , the Ceph Cluster should be same as k8s cluster.

@galexrt

This comment has been minimized.

Copy link
Member

galexrt commented Sep 28, 2018

@fkpwolf That the operator watches changes tot he CRD and/or Kubernetes nodes, is covered in #1435.

@galexrt

This comment has been minimized.

Copy link
Member

galexrt commented Oct 1, 2018

Closing as #1435 is tracking this feature/topic.

@galexrt galexrt closed this Oct 1, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment