Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GetNodeList API call returns empty cluster array #86

Closed
ltrilety opened this issue Feb 24, 2017 · 19 comments
Closed

GetNodeList API call returns empty cluster array #86

ltrilety opened this issue Feb 24, 2017 · 19 comments
Assignees
@r0h4n @shtripat
Labels
backend

Comments

@ltrilety
Copy link
Contributor

ltrilety commented Feb 24, 2017 •
edited
Loading

GetNodeList API call returns all nodes as it should but also it returns empty cluster array even when there exists some not imported cluster. Because of that tendrl frontend says: 'No Cluster Detected for Import.'.

Packages version:
tendrl-api-doc-1.2-02_24_2017_01_51_03.noarch
tendrl-api-1.2-02_24_2017_01_51_03.noarch
tendrl-api-httpd-1.2-02_24_2017_01_51_03.noarch
tendrl-dashboard-1.2-02_23_2017_11_56_35.noarch
tendrl-commons-1.2-02_24_2017_04_42_02.noarch
tendrl-node-agent-1.2-02_24_2017_00_01_02.noarch
tendrl-performance-monitoring-1.2-02_24_2017_03_02_02.noarch
tendrl-gluster-integration-1.2-02_24_2017_04_38_02.noarch
@ltrilety
Copy link
Contributor Author

ltrilety commented Feb 24, 2017 •
edited
Loading

example of GetNodeList API response:

{
"nodes": [
  {
"fqdn": "ltrilety-usm5-node4.usmqe.lab.eng.brq.redhat.com",
"machine_id": "56d65f493d8f854534fe792ecf014b83",
"node_id": "243cdc01-ce1b-484b-a317-1850bd66dba9",
...
},
  {
"machine_id": "56d65f493d8f854534fe792ecf014b83",
"node_id": "c4c6ea9d-f02f-4e6f-8176-6e72506004a9",
"fqdn": "ltrilety-usm5-node2.usmqe.lab.eng.brq.redhat.com",
...
},
  {
"node_id": "4db7b9bf-bf77-452a-b208-13fcbde05c43",
"fqdn": "ltrilety-usm5-node3.usmqe.lab.eng.brq.redhat.com",
"machine_id": "56d65f493d8f854534fe792ecf014b83",
...
},
  {
"node_id": "ff73b16b-4e9c-413c-9c06-69300a9de6ab",
"fqdn": "ltrilety-usm5-node1.usmqe.lab.eng.brq.redhat.com",
"machine_id": "56d65f493d8f854534fe792ecf014b83",
...
}
],
"clusters": [],
}

@ltrilety
Copy link
Contributor Author

ltrilety commented Feb 24, 2017 •
edited
Loading

tendrl-apid log:

-- Logs begin at Fri 2017-02-24 08:42:47 UTC, end at Fri 2017-02-24 10:51:17 UTC. --
Feb 24 09:07:20 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com systemd[1]: Started Tendrl Api Daemon.
Feb 24 09:07:20 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com systemd[1]: Starting Tendrl Api Daemon...
Feb 24 09:07:20 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: Puma starting in single mode...
Feb 24 09:07:20 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: * Version 3.6.0 (ruby 2.0.0-p648), codename: Sleepy Sunday Serenity
Feb 24 09:07:20 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: * Min threads: 0, max threads: 16
Feb 24 09:07:20 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: * Environment: production
Feb 24 09:07:20 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: * Listening on tcp://0.0.0.0:9292
Feb 24 09:07:20 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: Use Ctrl-C to stop
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: Etcd::KeyNotFound - Key not found:
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/etcd-0.3.0/lib/etcd/client.rb:141:in `process_http_request'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/etcd-0.3.0/lib/etcd/client.rb:114:in `api_execute'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/etcd-0.3.0/lib/etcd/keys.rb:22:in `get'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/tendrl-api/cluster.rb:7:in `block in <class:Cluster>'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1603:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1603:in `block in compile!'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `[]'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `block (3 levels) in route!'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:985:in `route_eval'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `block (2 levels) in route!'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1006:in `block in process_route'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1004:in `catch'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1004:in `process_route'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:964:in `block in route!'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:963:in `each'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:963:in `route!'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1076:in `block in dispatch!'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1073:in `dispatch!'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `block in call!'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `call!'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:886:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/logger.rb:15:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:210:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:180:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:2014:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1478:in `block in call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1788:in `synchronize'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1478:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:945:in `forward'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1019:in `route_missing'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:980:in `route!'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:976:in `route!'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:976:in `route!'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1076:in `block in dispatch!'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1073:in `dispatch!'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `block in call!'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `call!'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:886:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/logger.rb:15:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/commonlogger.rb:33:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:217:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:210:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:180:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:2014:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/puma-3.6.0/lib/puma/configuration.rb:225:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/puma-3.6.0/lib/puma/server.rb:578:in `handle_request'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/puma-3.6.0/lib/puma/server.rb:415:in `process_client'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/puma-3.6.0/lib/puma/server.rb:275:in `block in run'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/puma-3.6.0/lib/puma/thread_pool.rb:116:in `call'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/puma-3.6.0/lib/puma/thread_pool.rb:116:in `block in spawn_thread'
Feb 24 10:16:31 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: 10.40.204.171 - - [24/Feb/2017:10:16:31 +0000] "GET //1.0/GetClusterList HTTP/1.1" 404 35 0.0344
Feb 24 10:16:32 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: 10.40.204.171 - - [24/Feb/2017:10:16:32 +0000] "GET //1.0/GetNodeList HTTP/1.1" 200 56995 0.8451
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: Etcd::KeyNotFound - Key not found:
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/etcd-0.3.0/lib/etcd/client.rb:141:in `process_http_request'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/etcd-0.3.0/lib/etcd/client.rb:114:in `api_execute'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/etcd-0.3.0/lib/etcd/keys.rb:22:in `get'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/tendrl-api/cluster.rb:7:in `block in <class:Cluster>'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1603:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1603:in `block in compile!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `[]'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `block (3 levels) in route!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:985:in `route_eval'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `block (2 levels) in route!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1006:in `block in process_route'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1004:in `catch'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1004:in `process_route'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:964:in `block in route!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:963:in `each'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:963:in `route!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1076:in `block in dispatch!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1073:in `dispatch!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `block in call!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `call!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:886:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/logger.rb:15:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:210:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:180:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:2014:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1478:in `block in call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1788:in `synchronize'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1478:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:945:in `forward'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1019:in `route_missing'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:980:in `route!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:976:in `route!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:976:in `route!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1076:in `block in dispatch!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1073:in `dispatch!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `block in call!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `call!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:886:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/logger.rb:15:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/commonlogger.rb:33:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:217:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:210:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:180:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:2014:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/puma-3.6.0/lib/puma/configuration.rb:225:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/puma-3.6.0/lib/puma/server.rb:578:in `handle_request'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/puma-3.6.0/lib/puma/server.rb:415:in `process_client'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/puma-3.6.0/lib/puma/server.rb:275:in `block in run'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/puma-3.6.0/lib/puma/thread_pool.rb:116:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/puma-3.6.0/lib/puma/thread_pool.rb:116:in `block in spawn_thread'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: 10.40.204.171 - - [24/Feb/2017:10:16:54 +0000] "GET //1.0/GetClusterList HTTP/1.1" 404 35 0.0196
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: Etcd::KeyNotFound - Key not found:
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/etcd-0.3.0/lib/etcd/client.rb:141:in `process_http_request'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/etcd-0.3.0/lib/etcd/client.rb:114:in `api_execute'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/etcd-0.3.0/lib/etcd/keys.rb:22:in `get'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/tendrl-api/cluster.rb:7:in `block in <class:Cluster>'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1603:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1603:in `block in compile!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `[]'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `block (3 levels) in route!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:985:in `route_eval'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `block (2 levels) in route!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1006:in `block in process_route'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1004:in `catch'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1004:in `process_route'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:964:in `block in route!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:963:in `each'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:963:in `route!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1076:in `block in dispatch!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1073:in `dispatch!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `block in call!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `call!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:886:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/logger.rb:15:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:210:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:180:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:2014:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1478:in `block in call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1788:in `synchronize'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1478:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:945:in `forward'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1019:in `route_missing'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:980:in `route!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:976:in `route!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:976:in `route!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1076:in `block in dispatch!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1073:in `dispatch!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `block in call!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `call!'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:886:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/logger.rb:15:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/commonlogger.rb:33:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:217:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:210:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:180:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/sinatra-1.4.5/lib/sinatra/base.rb:2014:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/puma-3.6.0/lib/puma/configuration.rb:225:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/puma-3.6.0/lib/puma/server.rb:578:in `handle_request'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/puma-3.6.0/lib/puma/server.rb:415:in `process_client'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/puma-3.6.0/lib/puma/server.rb:275:in `block in run'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/puma-3.6.0/lib/puma/thread_pool.rb:116:in `call'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: /usr/share/gems/gems/puma-3.6.0/lib/puma/thread_pool.rb:116:in `block in spawn_thread'
Feb 24 10:16:54 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: 10.40.204.171 - - [24/Feb/2017:10:16:54 +0000] "GET //1.0/GetClusterList HTTP/1.1" 404 35 0.0208
Feb 24 10:17:04 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: 10.40.204.171 - - [24/Feb/2017:10:17:04 +0000] "GET //1.0/GetNodeList HTTP/1.1" 200 56995 1.1317
Feb 24 10:19:00 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: 10.40.204.171 - - [24/Feb/2017:10:19:00 +0000] "GET //1.0/GetNodeList HTTP/1.1" 200 56995 0.9437
Feb 24 10:19:12 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: 10.40.204.223 - - [24/Feb/2017:10:19:12 +0000] "GET //1.0/GetNodeList HTTP/1.1" 200 56995 0.8906
Feb 24 10:24:02 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: 10.40.204.171 - - [24/Feb/2017:10:24:02 +0000] "GET //1.0/GetNodeList HTTP/1.1" 200 57032 0.8857
Feb 24 10:41:11 ltrilety-usm5-server.usmqe.lab.eng.brq.redhat.com puma[21215]: 10.40.204.171 - - [24/Feb/2017:10:41:11 +0000] "GET //1.0/GetNodeList HTTP/1.1" 200 56679 0.7541

@ltrilety ltrilety mentioned this issue Feb 24, 2017
Closed
@sankarshanmukhopadhyay
Copy link
Contributor

@anivargi please confirm whether this is an API topic or, should be assigned to back-end

@r0h4n
Copy link
Contributor

r0h4n commented Feb 28, 2017

@ltrilety Please provide logs for all tendrl services sorted by nodes in a tar file

@ltrilety
Copy link
Contributor Author

ltrilety commented Mar 1, 2017

Logs and configuration files from my setup
tendrl.tar.gz

@anivargi
Copy link
Contributor

anivargi commented Mar 2, 2017

@ltrilety the clusters array returns the available clusters which can be imported, if you have already imported all the detected clusters, it will be empty.

@sankarshanmukhopadhyay
Copy link
Contributor

@anivargi in other words, are you stating that it is designed to be such and behaving as expected?

@anivargi
Copy link
Contributor

anivargi commented Mar 3, 2017

@sankarshanmukhopadhyay yes

@ltrilety
Copy link
Contributor Author

ltrilety commented Mar 3, 2017 •
edited
Loading

@anivargi that's not the case as I said in my first comment: "it returns empty cluster array even when there exists some not imported cluster".
In other words no cluster was imported/present in tendrl none the less cluster list in the response was empty.

@anivargi
Copy link
Contributor

anivargi commented Mar 8, 2017

@ltrilety The issue is that the nodes don't have detected cluster inside of it hence the clusters come up as empty.
@r0h4n @shtripat Please look into it.

@anivargi anivargi assigned r0h4n and shtripat and unassigned anivargi Mar 8, 2017
@shtripat
Copy link
Member

shtripat commented Mar 8, 2017

Tested with latest node-agent master for gluster cluster.It works fine and DetectedCluster field is getting populated as below

{

    "action": "get",
    "node": {
        "key": "/nodes/9874219c-2142-45c5-8d94-93f0f43b7dde/DetectedCluster",
        "dir": true,
        "nodes": [
            {
                "key": "/nodes/9874219c-2142-45c5-8d94-93f0f43b7dde/DetectedCluster/detected_cluster_id",
                "value": "eacb7101e388754099783568dc5af386c38abb3faee4083e9bb7a6e399c6f365",
                "modifiedIndex": 799,
                "createdIndex": 799
            },
            {
                "key": "/nodes/9874219c-2142-45c5-8d94-93f0f43b7dde/DetectedCluster/sds_pkg_name",
                "value": "glusterfs",
                "modifiedIndex": 800,
                "createdIndex": 800
            },
            {
                "key": "/nodes/9874219c-2142-45c5-8d94-93f0f43b7dde/DetectedCluster/sds_pkg_version",
                "value": "3.9.0",
                "modifiedIndex": 801,
                "createdIndex": 801
            }
        ],
        "modifiedIndex": 799,
        "createdIndex": 799
    }
}

For ceph cluster nodes, will update soon.

@shtripat
Copy link
Member

shtripat commented Mar 8, 2017

For ceph cluster as well it does list the DetectedCluster field for nodes as below with latest master node-agent

{

    "action": "get",
    "node": {
        "key": "/nodes/bafc7e0b-9ad0-4e2b-8e4f-f5328c4ed6f3/DetectedCluster",
        "dir": true,
        "nodes": [
            {
                "key": "/nodes/bafc7e0b-9ad0-4e2b-8e4f-f5328c4ed6f3/DetectedCluster/sds_pkg_version",
                "value": "10.2.5",
                "modifiedIndex": 7766,
                "createdIndex": 7766
            },
            {
                "key": "/nodes/bafc7e0b-9ad0-4e2b-8e4f-f5328c4ed6f3/DetectedCluster/detected_cluster_id",
                "value": "c221ccdb-51d6-4b57-9f10-bcf30c7fa351",
                "modifiedIndex": 7764,
                "createdIndex": 7764
            },
            {
                "key": "/nodes/bafc7e0b-9ad0-4e2b-8e4f-f5328c4ed6f3/DetectedCluster/sds_pkg_name",
                "value": "ceph",
                "modifiedIndex": 7765,
                "createdIndex": 7765
            }
        ],
        "modifiedIndex": 7764,
        "createdIndex": 7764
    }
}

@shtripat
Copy link
Member

shtripat commented Mar 8, 2017 •
edited
Loading

@fbalak I tried debugging the QE set as well and all I have done is that re-started tendrl-node-agent service on all the storage nodes and I see DetectedCluster field listed for all the nodes now.
Also GetNodeList api is working and returning details properly (check http://{srvr node}:9292/1.0/GetNodeList).

So effectively looks like the service tendrl-node-agent was not re-started post update rpms or something.

PS: Yes to re-start I had to do service tendrl-node-agent stop and service tendrl-node-agent start one by one, so I feel there could be some issues around service tendrl-node-agent restart command which could be debugged under different issue I feel.
For the time being use stop+start if there is an update of rpms.

fbalak added a commit to fbalak/usmqe-setup that referenced this issue Mar 9, 2017
@fbalak
workaround for Tendrl/api#86
0509fba 
based on:
Tendrl/api#86 (comment)
TODO: remove this after issue resolution
@fbalak fbalak mentioned this issue Mar 9, 2017
Merged
@fbalak
Copy link
Contributor

fbalak commented Mar 9, 2017

@shtripat Restarting tendrl-node-agent service helps but only after all components are installed and gluster trusted pool is created. Restarting tendrl-node-agent service right after tendrl-node-agent installation doesn't work. Tested with fresh installation:
tendrl-commons-1.2.1-03_09_2017_05_04_44.noarch
tendrl-node-agent-1.2.1-03_09_2017_00_01_05.noarch
tendrl-gluster-integration-1.2.1-03_09_2017_04_39_14.noarch

@shtripat
Copy link
Member

shtripat commented Mar 9, 2017

@fbalak so in the case there are no storage bits on the nodes (niether ceph nor gluster) certainly detected cluster details would be empty.
The DetectedCluster field is useful for import cluster flow where we know that at least ceph or gluster bits are installed on the nodes and in that case DetectedCluster should be present for node details.

@anivargi now if API alway depends on DetectedCluster field for listing the nodes, that could be an issue for create cluster flow as in that case there wont be storage bits installed on the nodes readily and as part of create cluster flow that would happen. May be a fix required in API layer for this.

@fbalak
Copy link
Contributor

fbalak commented Mar 9, 2017

@shtripat Yes, I just meant that I restarted tendrl-node-agent right after installation of node-agent and checked GetNodeList after trusted pool was created and it returned no importable cluster. I had to restart it again after installation of all components to make it work so maybe some of those components breaks it.

fbalak added a commit to fbalak/usmqe-setup that referenced this issue Mar 9, 2017
@fbalak
add workaround for Tendrl/api#86
f45cb68 
workaround according to Tendrl/api#86 (comment)
TODO: delete after resolving
mbukatov added a commit to usmqe/usmqe-setup that referenced this issue Mar 9, 2017
@mbukatov
Merge pull request #57 from fbalak/na-workaround
6ae63d5 
workaround for Tendrl/api#86
@mbukatov mbukatov mentioned this issue Mar 9, 2017
Closed
@anivargi anivargi removed the API label Mar 14, 2017
@r0h4n
Copy link
Contributor

r0h4n commented Apr 4, 2017

Is this still reproducible?

@fbalak
Copy link
Contributor

fbalak commented Apr 4, 2017

@ltrilety you can close this. Seems ok.
Tested with:
tendrl-api-1.2.1-04_04_2017_01_51_03.noarch
tendrl-commons-1.2.1-04_01_2017_00_59_17.noarch
tendrl-node-agent-1.2.2-04_04_2017_00_01_04.noarch
tendrl-gluster-integration-1.2.1-04_04_2017_04_38_03.noarch

@ltrilety
Copy link
Contributor Author

ltrilety commented Apr 4, 2017

@fbalak ok, thanks for testing that

@ltrilety ltrilety closed this as completed Apr 4, 2017
fbalak added a commit to fbalak/usmqe-setup that referenced this issue Apr 4, 2017
@fbalak
remove tasks related to workaround
f60ad0b 
Tendrl/api#86
@fbalak fbalak mentioned this issue Apr 4, 2017
Merged
@ltrilety ltrilety mentioned this issue Apr 5, 2017
Closed
fbalak added a commit to fbalak/usmqe-setup that referenced this issue May 25, 2017
@fbalak
remove workaround
2ec99fb 
Tendrl/api#86 (comment)
@fbalak fbalak mentioned this issue May 25, 2017
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@r0h4n r0h4n

@shtripat shtripat

Labels
backend
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@r0h4n @sankarshanmukhopadhyay @shtripat @anivargi @fbalak @ltrilety