Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot create cluster #453

Closed
mkudlej opened this issue May 19, 2017 · 10 comments
Closed

Cannot create cluster #453

mkudlej opened this issue May 19, 2017 · 10 comments
Assignees
@GowthamShanmugam

Comments

@mkudlej
Copy link

mkudlej commented May 19, 2017

I'm not able to create cluster via API because of:

{"errors":{"missing":"Unavailable nodes: 10.70.16.146, 10.70.16.148, 10.70.16.168, 10.70.16.176, 10.70.16.186, 10.70.16.197, 10.70.16.207."}}

which means according code that there is no

/indexes/ip

in etcd.
I've tried to look at /networks/ in etcd according #400 (comment)
But there is no /networks/.

Maybe problem is that ethernet device's name is ens3:

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:e1:ec:61 brd ff:ff:ff:ff:ff:ff
    inet 10.70.16.166/24 brd 10.70.16.255 scope global dynamic ens3
       valid_lft 41465sec preferred_lft 41465sec
    inet6 fe80::5054:ff:fee1:ec61/64 scope link 
       valid_lft forever preferred_lft forever

We need this interface because we would like to see statistics in Tendrl. Ask @dahorak for details.

I don't see any errors related to network in logs.
Used packages:

centos-release-ceph-jewel-1.0-1.el7.centos.noarch
ceph-ansible-2.2.0-139.gfa168e9.el7.noarch
ceph-installer-1.3.0-10.g0e9fec2.el7.noarch
etcd-3.1.3-1.el7.x86_64
python-etcd-0.4.5-1.noarch
rubygem-etcd-0.3.0-1.el7.centos.noarch
tendrl-api-1.3.0-05_19_2017_01_51_03.noarch
tendrl-api-doc-1.3.0-05_19_2017_01_51_03.noarch
tendrl-api-httpd-1.3.0-05_19_2017_01_51_03.noarch
tendrl-commons-1.3.0-05_19_2017_05_13_06.noarch
tendrl-dashboard-1.3.0-05_19_2017_04_11_04.noarch
tendrl-node-agent-1.3.0-05_19_2017_00_01_05.noarch
tendrl-node-monitoring-1.3.0-05_19_2017_04_27_26.noarch
tendrl-performance-monitoring-1.3.0-05_19_2017_03_02_04.noarch
@GowthamShanmugam
Copy link
Contributor

#454

@GowthamShanmugam
Copy link
Contributor

This issue is solved

@r0h4n
Copy link
Contributor

r0h4n commented May 19, 2017

This was related to failure in parsing Disk inventory which caused the node to not be available. We have fixed this, please try with latest release rpms

@japplewhite
Copy link

@r0h4n Still not seeing nodes show up in the UI with this fix, although I do see /indexes/ip now.

tendrl-commons-1.3.0-05_21_2017_02_01_09.noarch
tendrl-node-agent-1.3.0-05_21_2017_02_39_12.noarch
tendrl-api-1.3.0-05_21_2017_01_51_03.noarch
tendrl-api-httpd-1.3.0-05_21_2017_01_51_03.noarch
tendrl-performance-monitoring-1.3.0-05_21_2017_03_02_05.noarch
tendrl-api-doc-1.3.0-05_21_2017_01_51_03.noarch
tendrl-dashboard-1.3.0-05_20_2017_04_11_04.noarch

[root@tendrl ~]# curl -XGET http://10.136.63.106:2379/v2/keys/indexes/ip
{"action":"get","node":{"key":"/indexes/ip","dir":true,"nodes":[{"key":"/indexes/ip/10.10.0.7","value":"5ecb4734-47a1-4dc3-bf5d-0fae5f0d1779","modifiedIndex":1499,"createdIndex":1499},{"key":"/indexes/ip/67.207.90.99","value":"d3b94f2d-acdf-42ff-b4d2-8ee5ad287627","modifiedIndex":2214,"createdIndex":2214},{"key":"/indexes/ip/10.136.63.211","value":"f63fcf98-0b7b-4544-94e4-1e2912b2b96d","modifiedIndex":2813,"createdIndex":2813},{"key":"/indexes/ip/10.136.63.212","value":"19bc4da3-259e-4978-ade2-d9f537de927c","modifiedIndex":2876,"createdIndex":2876},{"key":"/indexes/ip/10.10.0.12","value":"99bebc7a-bfb6-452c-ac7c-8fdf63ecaf66","modifiedIndex":2590,"createdIndex":2590},{"key":"/indexes/ip/192.241.147.13","value":"f63fcf98-0b7b-4544-94e4-1e2912b2b96d","modifiedIndex":2833,"createdIndex":2833},{"key":"/indexes/ip/67.207.90.79","value":"5ecb4734-47a1-4dc3-bf5d-0fae5f0d1779","modifiedIndex":1498,"createdIndex":1498},{"key":"/indexes/ip/10.136.50.211","value":"e912b8c6-36a7-415e-9845-d9a67bcc32c3","modifiedIndex":1549,"createdIndex":1549},{"key":"/indexes/ip/10.136.63.224","value":"7af3686e-3261-450e-bcd1-f1c7a5aaddff","modifiedIndex":2300,"createdIndex":2300},{"key":"/indexes/ip/10.10.0.8","value":"d3b94f2d-acdf-42ff-b4d2-8ee5ad287627","modifiedIndex":2215,"createdIndex":2215},{"key":"/indexes/ip/10.136.63.106","value":"2d9b7ad6-57fd-4f71-974c-492afb1b6533","modifiedIndex":270,"createdIndex":270},{"key":"/indexes/ip/10.10.0.5","value":"2d9b7ad6-57fd-4f71-974c-492afb1b6533","modifiedIndex":290,"createdIndex":290},{"key":"/indexes/ip/10.136.63.200","value":"5ecb4734-47a1-4dc3-bf5d-0fae5f0d1779","modifiedIndex":1475,"createdIndex":1475},{"key":"/indexes/ip/198.199.77.225","value":"e912b8c6-36a7-415e-9845-d9a67bcc32c3","modifiedIndex":1628,"createdIndex":1628},{"key":"/indexes/ip/10.10.0.6","value":"e912b8c6-36a7-415e-9845-d9a67bcc32c3","modifiedIndex":1629,"createdIndex":1629},{"key":"/indexes/ip/67.207.90.117","value":"19bc4da3-259e-4978-ade2-d9f537de927c","modifiedIndex":2915,"createdIndex":2915},{"key":"/indexes/ip/10.136.62.214","value":"2fb8031b-7a95-4971-960f-45fe94fc3e38","modifiedIndex":2140,"createdIndex":2140},{"key":"/indexes/ip/10.10.0.9","value":"2fb8031b-7a95-4971-960f-45fe94fc3e38","modifiedIndex":2205,"createdIndex":2205},{"key":"/indexes/ip/10.136.63.220","value":"99bebc7a-bfb6-452c-ac7c-8fdf63ecaf66","modifiedIndex":2502,"createdIndex":2502},{"key":"/indexes/ip/67.207.90.127","value":"7af3686e-3261-450e-bcd1-f1c7a5aaddff","modifiedIndex":2356,"createdIndex":2356},{"key":"/indexes/ip/67.207.90.124","value":"99bebc7a-bfb6-452c-ac7c-8fdf63ecaf66","modifiedIndex":2589,"createdIndex":2589},{"key":"/indexes/ip/10.10.0.10","value":"f63fcf98-0b7b-4544-94e4-1e2912b2b96d","modifiedIndex":2834,"createdIndex":2834},{"key":"/indexes/ip/10.136.63.203","value":"d3b94f2d-acdf-42ff-b4d2-8ee5ad287627","modifiedIndex":2160,"createdIndex":2160},{"key":"/indexes/ip/10.10.0.13","value":"7af3686e-3261-450e-bcd1-f1c7a5aaddff","modifiedIndex":2357,"createdIndex":2357},{"key":"/indexes/ip/10.10.0.11","value":"19bc4da3-259e-4978-ade2-d9f537de927c","modifiedIndex":2916,"createdIndex":2916},{"key":"/indexes/ip/67.205.179.34","value":"2d9b7ad6-57fd-4f71-974c-492afb1b6533","modifiedIndex":289,"createdIndex":289},{"key":"/indexes/ip/67.207.90.101","value":"2fb8031b-7a95-4971-960f-45fe94fc3e38","modifiedIndex":2203,"createdIndex":2203}],"modifiedIndex":270,"createdIndex":270}}

[root@tendrl ~]# curl -XGET http://10.136.63.106:2379/v2/keys/nodes
{"action":"get","node":{"key":"/nodes","dir":true,"nodes":[{"key":"/nodes/e912b8c6-36a7-415e-9845-d9a67bcc32c3","dir":true,"modifiedIndex":544,"createdIndex":544},{"key":"/nodes/2d9b7ad6-57fd-4f71-974c-492afb1b6533","dir":true,"modifiedIndex":8,"createdIndex":8},{"key":"/nodes/5ecb4734-47a1-4dc3-bf5d-0fae5f0d1779","dir":true,"modifiedIndex":446,"createdIndex":446},{"key":"/nodes/f63fcf98-0b7b-4544-94e4-1e2912b2b96d","dir":true,"modifiedIndex":497,"createdIndex":497},{"key":"/nodes/7af3686e-3261-450e-bcd1-f1c7a5aaddff","dir":true,"modifiedIndex":579,"createdIndex":579},{"key":"/nodes/99bebc7a-bfb6-452c-ac7c-8fdf63ecaf66","dir":true,"modifiedIndex":592,"createdIndex":592},{"key":"/nodes/2fb8031b-7a95-4971-960f-45fe94fc3e38","dir":true,"modifiedIndex":481,"createdIndex":481},{"key":"/nodes/d3b94f2d-acdf-42ff-b4d2-8ee5ad287627","dir":true,"modifiedIndex":510,"createdIndex":510},{"key":"/nodes/19bc4da3-259e-4978-ade2-d9f537de927c","dir":true,"modifiedIndex":530,"createdIndex":530}],"modifiedIndex":8,"createdIndex":8}}

@r0h4n
Copy link
Contributor

r0h4n commented May 21, 2017

Hey Jeff, can you please Pastebin all the detail data for nodes/:node_id/*

@cloudbehl @anivargi @GowthamShanmugam please take a look

@japplewhite
Copy link

@r0h4n @cloudbehl @anivargi @GowthamShanmugam
https://pastebin.com/0XyYn2u3

@mkudlej
Copy link
Author

mkudlej commented May 22, 2017

I still see this issue with:

centos-release-ceph-jewel-1.0-1.el7.centos.noarch
ceph-ansible-2.2.0-141.ge605445.el7.noarch
ceph-installer-1.3.0-10.g0e9fec2.el7.noarch
etcd-3.1.3-1.el7.x86_64
python-etcd-0.4.5-1.noarch
rubygem-etcd-0.3.0-1.el7.centos.noarch
tendrl-api-1.3.0-05_22_2017_01_51_04.noarch
tendrl-api-doc-1.3.0-05_22_2017_01_51_04.noarch
tendrl-api-httpd-1.3.0-05_22_2017_01_51_04.noarch
tendrl-commons-1.3.0-05_22_2017_05_13_04.noarch
tendrl-dashboard-1.3.0-05_22_2017_04_11_04.noarch
tendrl-node-agent-1.3.0-05_22_2017_00_01_06.noarch
tendrl-node-monitoring-1.3.0-05_22_2017_04_25_03.noarch
tendrl-performance-monitoring-1.3.0-05_22_2017_03_02_07.noarch

@mkudlej
Copy link
Author

mkudlej commented May 22, 2017

And indexes are there:

$ curl -XGET http://mkudlej-usm2-server.usmqe.lab.eng.blr.redhat.com:2379/v2/keys/indexes/ip
{"action":"get","node":{"key":"/indexes/ip","dir":true,"nodes":[{"key":"/indexes/ip/10.70.16.146","value":"acf9c805-143b-4660-8618-6ddfdbeff2cb","modifiedIndex":3817,"createdIndex":3817},{"key":"/indexes/ip/10.70.16.148","value":"d4e21516-634a-478b-a6ae-6170279888ed","modifiedIndex":3930,"createdIndex":3930},{"key":"/indexes/ip/10.70.16.168","value":"a0884100-42e6-45bd-897d-1eefa9e231d4","modifiedIndex":3883,"createdIndex":3883}],"modifiedIndex":3817,"createdIndex":3817}}

@r0h4n
Copy link
Contributor

r0h4n commented May 22, 2017 •
edited

Folks, the tendrl-dashboard was unable to parse the fixed disk inventory data coming out of our nodes

This PR (Tendrl/ui#357) fixes it.

Could you guys give it a spin again using latest release rpms

@mkudlej
Copy link
Author

mkudlej commented May 23, 2017

It seems to work with

centos-release-ceph-jewel-1.0-1.el7.centos.noarch
ceph-ansible-2.2.6-1.el7scon.noarch
ceph-base-11.2.0-0.el7.x86_64
ceph-common-11.2.0-0.el7.x86_64
ceph-installer-1.3.0-1.el7scon.noarch
ceph-mon-11.2.0-0.el7.x86_64
ceph-osd-11.2.0-0.el7.x86_64
ceph-selinux-11.2.0-0.el7.x86_64
etcd-3.1.3-1.el7.x86_64
glusterfs-3.9.1-1.el7.x86_64
glusterfs-api-3.9.1-1.el7.x86_64
glusterfs-cli-3.9.1-1.el7.x86_64
glusterfs-client-xlators-3.9.1-1.el7.x86_64
glusterfs-fuse-3.9.1-1.el7.x86_64
glusterfs-libs-3.9.1-1.el7.x86_64
glusterfs-server-3.9.1-1.el7.x86_64
libcephfs2-11.2.0-0.el7.x86_64
python-cephfs-11.2.0-0.el7.x86_64
python-etcd-0.4.5-1.noarch
rubygem-etcd-0.3.0-1.el7.centos.noarch
tendrl-api-1.3.0-05_22_2017_01_51_04.noarch
tendrl-api-doc-1.3.0-05_22_2017_01_51_04.noarch
tendrl-api-httpd-1.3.0-05_22_2017_01_51_04.noarch
tendrl-commons-1.3.0-05_22_2017_20_33_47.noarch
tendrl-dashboard-1.3.0-05_22_2017_19_15_06.noarch
tendrl-gluster-integration-1.3.0-05_23_2017_06_23_06.noarch
tendrl-node-agent-1.3.0-05_22_2017_19_20_55.noarch
tendrl-node-monitoring-1.3.0-05_22_2017_04_25_03.noarch
tendrl-performance-monitoring-1.3.0-05_22_2017_03_02_07.noarch

@mkudlej mkudlej closed this as completed May 23, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@GowthamShanmugam GowthamShanmugam

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@r0h4n @mkudlej @japplewhite @GowthamShanmugam