Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

_cat/nodes causes NullPointerException in 1.2.0 #6297

Closed
avleen opened this issue May 23, 2014 · 24 comments
Closed

_cat/nodes causes NullPointerException in 1.2.0 #6297

avleen opened this issue May 23, 2014 · 24 comments
Assignees
@imotov
Labels
>bug

Comments

@avleen
Copy link

avleen commented May 23, 2014

curl -s 'http://localhost:9200/_cat/nodes'
{"error":"NullPointerException[null]","status":500}

Nothing else is logged on the masters or data nodes.
We have Logstash 1.4.1 connected to the cluster, but this happens regardless of whether the Logstash nodes are connected or not.
@drewr drewr self-assigned this May 23, 2014
@drewr
Copy link
Contributor

drewr commented May 23, 2014

I can't repro locally. Can you post /_cluster/state?filter_metadata and /_nodes?all somewhere?

@s1monw
Copy link
Contributor

s1monw commented May 23, 2014

or do you see a stacktrace in the logs somewhere?

@avleen
Copy link
Author

avleen commented May 23, 2014

Hi @drewr ew, I hope you don't mind, I emailed them to you as they contain data I'd prefer not to post publicly :)

@s1monw no stack traces from this in any logs on any ES node.

@drewr
Copy link
Contributor

drewr commented May 23, 2014

@avleen I don't mind at all. However, I haven't seen anything yet. Where did you send it? You can use first.last@elasticsearch.

Update: Nevermind, spam. 😈

@avleen
Copy link
Author

avleen commented May 23, 2014

Hi Drew,

Just double checked.. Yup that's where I sent it, at 17:29 ET.
Has two attachments, a zip file and a text file.
On May 23, 2014 6:32 PM, "Drew Raines" notifications@github.com wrote:

@avleen https://github.com/avleen I don't mind at all. However, I
haven't seen anything yet. You can use first.last@elasticsearch.

Reply to this email directly or view it on GitHubhttps://github.com//issues/6297#issuecomment-44067235
.

@mcleancraig
Copy link

FWIW I'm also seeing this, on a freshly-updated 1.2.1 cluster (from 1.1.1) with 64-bit Java 1.7.0_55, CentOS 5.8.

@martijnvg
Copy link
Member

This PR addresses should fix this NullPointerException:
#6190

The fix will be included in 1.2.2

@mcnewton
Copy link

Hi,
For info - just installed ES for the first time a week ago (1.2.1 debian wheezy 64bit), and hit this issue. Upgraded to 1.2.2 yesterday, and it still reports the NullPointerException when requesting _cat/nodes. Installed using the debian packages. Also get on a newly installed 1.2.2 node (though joined to the same cluster).
Cheers,
Matthew

@dadoonet
Copy link
Member

Any stack trace in nodes logs?

@mcnewton
Copy link

No, nothing in node logs. Is there any way to increase debugging, maybe? I wondered if the package hadn't upgraded correctly, but it's all reporting 1.2.2 (logs say version[1.2.2], pid[10140], build[9902f08/2014-07-09T12:02:32Z]).

@faxm0dem
Copy link

I can confirm this bug is still present in 1.3.0, even after upgrading all nodes:

$ curl 0:9200

{
  "status" : 200,
  "name" : "mycluster",
  "version" : {
    "number" : "1.3.0",
    "build_hash" : "1265b1454eee7725a6918f57415c480028700fb4",
    "build_timestamp" : "2014-07-23T13:46:36Z",
    "build_snapshot" : false,
    "lucene_version" : "4.9"
  },
  "tagline" : "You Know, for Search"
}
$ curl 0:9200/_cat/nodes

{"error":"NullPointerException[null]","status":500}

Nothing in the logfile.

@clintongormley
Copy link

@faxm0dem do you have multiple nodes running? if so did you look in the logs for all the nodes?

could you send the output of:

curl 0:9200/_nodes
curl 0:9200/_nodes/stats

thanks

@faxm0dem
Copy link

Yes, two nodes, nothing on either side.
Here's the output:

curl 0:9200/_nodes

{
  "cluster_name" : "telecom",
  "nodes" : {
    "JUOZAr-mT6e8e9nmwWH9ww" : {
      "name" : "node07-telecom",
      "transport_address" : "inet[/10.0.104.214:9300]",
      "host" : "node07",
      "ip" : "10.0.104.214",
      "version" : "1.3.0",
      "build" : "1265b14",
      "http_address" : "inet[/10.0.104.214:9200]",
      "settings" : {
        "node" : {
          "name" : "node07-telecom"
        },
        "index" : {
          "number_of_replicas" : "1",
          "number_of_shards" : "8"
        },
        "bootstrap" : {
          "mlockall" : "true"
        },
        "name" : "node07-telecom",
        "pidfile" : "/var/run/elasticsearch/elasticsearch-telecom.pid",
        "path" : {
          "data" : "/var/lib/elasticsearch/telecom",
          "work" : "/tmp/elasticsearch",
          "home" : "/usr/share/elasticsearch",
          "conf" : "/etc/elasticsearch/telecom",
          "logs" : "/var/log/elasticsearch/telecom"
        },
        "cluster" : {
          "name" : "telecom"
        },
        "indices" : {
          "memory" : {
            "index_buffer_size" : "30%"
          }
        },
        "discovery" : {
          "zen" : {
            "minimum_master_nodes" : "1",
            "ping" : {
              "unicast" : {
                "hosts" : [ "node07", "node38" ]
              },
              "multicast" : {
                "enabled" : "false"
              },
              "timeout" : "30s"
            }
          }
        }
      },
      "os" : {
        "refresh_interval_in_millis" : 1000,
        "available_processors" : 16,
        "cpu" : {
          "vendor" : "Intel",
          "model" : "Xeon",
          "mhz" : 2527,
          "total_cores" : 16,
          "total_sockets" : 1,
          "cores_per_socket" : 16,
          "cache_size_in_bytes" : 8192
        },
        "mem" : {
          "total_in_bytes" : 25185079296
        },
        "swap" : {
          "total_in_bytes" : 279650304
        }
      },
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 27129,
        "max_file_descriptors" : 65535,
        "mlockall" : false
      },
      "jvm" : {
        "pid" : 27129,
        "version" : "1.7.0_65",
        "vm_name" : "OpenJDK 64-Bit Server VM",
        "vm_version" : "24.65-b04",
        "vm_vendor" : "Oracle Corporation",
        "start_time_in_millis" : 1406268341800,
        "mem" : {
          "heap_init_in_bytes" : 17179869184,
          "heap_max_in_bytes" : 17066491904,
          "non_heap_init_in_bytes" : 24313856,
          "non_heap_max_in_bytes" : 224395264,
          "direct_max_in_bytes" : 17066491904
        },
        "gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
        "memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
      },
      "thread_pool" : {
        "generic" : {
          "type" : "cached",
          "keep_alive" : "30s",
          "queue_size" : -1
        },
        "index" : {
          "type" : "fixed",
          "min" : 16,
          "max" : 16,
          "queue_size" : "200"
        },
        "snapshot_data" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "bench" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "get" : {
          "type" : "fixed",
          "min" : 16,
          "max" : 16,
          "queue_size" : "1k"
        },
        "snapshot" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "merge" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "suggest" : {
          "type" : "fixed",
          "min" : 16,
          "max" : 16,
          "queue_size" : "1k"
        },
        "bulk" : {
          "type" : "fixed",
          "min" : 16,
          "max" : 16,
          "queue_size" : "50"
        },
        "optimize" : {
          "type" : "fixed",
          "min" : 1,
          "max" : 1,
          "queue_size" : -1
        },
        "warmer" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "flush" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "search" : {
          "type" : "fixed",
          "min" : 48,
          "max" : 48,
          "queue_size" : "1k"
        },
        "percolate" : {
          "type" : "fixed",
          "min" : 16,
          "max" : 16,
          "queue_size" : "1k"
        },
        "management" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "refresh" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 8,
          "keep_alive" : "5m",
          "queue_size" : -1
        }
      },
      "network" : {
        "refresh_interval_in_millis" : 5000,
        "primary_interface" : {
          "address" : "10.0.104.214",
          "name" : "eth0",
          "mac_address" : ""
        }
      },
      "transport" : {
        "bound_address" : "inet[/0:0:0:0:0:0:0:0%0:9300]",
        "publish_address" : "inet[/10.0.104.214:9300]"
      },
      "http" : {
        "bound_address" : "inet[/0:0:0:0:0:0:0:0%0:9200]",
        "publish_address" : "inet[/10.0.104.214:9200]",
        "max_content_length_in_bytes" : 104857600
      },
      "plugins" : [ ]
    },
    "JL38-jS9Sn67hQk2XxNZxw" : {
      "name" : "logstash-netflow",
      "transport_address" : "inet[/10.0.108.171:9303]",
      "host" : "node38",
      "ip" : "10.0.108.171",
      "version" : "1.1.1",
      "build" : "f1585f0",
      "attributes" : {
        "client" : "true",
        "data" : "false"
      },
      "settings" : {
        "path" : {
          "logs" : "/home/sysunix/logs"
        },
        "cluster" : {
          "name" : "telecom"
        },
        "node" : {
          "client" : "true",
          "name" : "logstash-netflow"
        },
        "discovery" : {
          "zen" : {
            "ping" : {
              "unicast" : {
                "hosts" : "node38:9300,node38:9301,node38:9302,node38:9303,node38:9304,node38:9305"
              },
              "multicast" : {
                "enabled" : "false"
              }
            }
          }
        },
        "http" : {
          "enabled" : "false"
        },
        "name" : "logstash-netflow"
      },
      "os" : {
        "refresh_interval_in_millis" : 1000,
        "available_processors" : 32
      },
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 8365,
        "max_file_descriptors" : 4096,
        "mlockall" : false
      },
      "jvm" : {
        "pid" : 8365,
        "version" : "1.7.0_55",
        "vm_name" : "OpenJDK 64-Bit Server VM",
        "vm_version" : "24.51-b03",
        "vm_vendor" : "Oracle Corporation",
        "start_time_in_millis" : 1402068863618,
        "mem" : {
          "heap_init_in_bytes" : 524288000,
          "heap_max_in_bytes" : 506855424,
          "non_heap_init_in_bytes" : 24313856,
          "non_heap_max_in_bytes" : 224395264,
          "direct_max_in_bytes" : 506855424
        },
        "gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
        "memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
      },
      "thread_pool" : {
        "generic" : {
          "type" : "cached",
          "keep_alive" : "30s",
          "queue_size" : -1
        },
        "index" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "200"
        },
        "get" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "1k"
        },
        "snapshot" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "merge" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "suggest" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "1k"
        },
        "bulk" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "50"
        },
        "optimize" : {
          "type" : "fixed",
          "min" : 1,
          "max" : 1,
          "queue_size" : -1
        },
        "warmer" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "flush" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "search" : {
          "type" : "fixed",
          "min" : 96,
          "max" : 96,
          "queue_size" : "1k"
        },
        "percolate" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "1k"
        },
        "management" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "refresh" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 10,
          "keep_alive" : "5m",
          "queue_size" : -1
        }
      },
      "network" : {
        "refresh_interval_in_millis" : 5000,
        "primary_interface" : {
          "address" : "",
          "name" : "",
          "mac_address" : ""
        }
      },
      "transport" : {
        "bound_address" : "inet[/0:0:0:0:0:0:0:0%0:9303]",
        "publish_address" : "inet[/10.0.108.171:9303]"
      },
      "plugins" : [ ]
    },
    "rPZ9EsahRl-9Cs_AOucOJQ" : {
      "name" : "logstash-netflow",
      "transport_address" : "inet[/10.0.108.171:9302]",
      "host" : "node38",
      "ip" : "10.0.108.171",
      "version" : "1.1.1",
      "build" : "f1585f0",
      "attributes" : {
        "client" : "true",
        "data" : "false"
      },
      "settings" : {
        "path" : {
          "logs" : "/home/sysunix/logs"
        },
        "cluster" : {
          "name" : "telecom"
        },
        "node" : {
          "client" : "true",
          "name" : "logstash-netflow"
        },
        "discovery" : {
          "zen" : {
            "ping" : {
              "unicast" : {
                "hosts" : "node38:9300,node38:9301,node38:9302,node38:9303,node38:9304,node38:9305"
              },
              "multicast" : {
                "enabled" : "false"
              }
            }
          }
        },
        "http" : {
          "enabled" : "false"
        },
        "name" : "logstash-netflow"
      },
      "os" : {
        "refresh_interval_in_millis" : 1000,
        "available_processors" : 32
      },
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 8365,
        "max_file_descriptors" : 4096,
        "mlockall" : false
      },
      "jvm" : {
        "pid" : 8365,
        "version" : "1.7.0_55",
        "vm_name" : "OpenJDK 64-Bit Server VM",
        "vm_version" : "24.51-b03",
        "vm_vendor" : "Oracle Corporation",
        "start_time_in_millis" : 1402068863618,
        "mem" : {
          "heap_init_in_bytes" : 524288000,
          "heap_max_in_bytes" : 506855424,
          "non_heap_init_in_bytes" : 24313856,
          "non_heap_max_in_bytes" : 224395264,
          "direct_max_in_bytes" : 506855424
        },
        "gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
        "memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
      },
      "thread_pool" : {
        "generic" : {
          "type" : "cached",
          "keep_alive" : "30s",
          "queue_size" : -1
        },
        "index" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "200"
        },
        "get" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "1k"
        },
        "snapshot" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "merge" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "suggest" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "1k"
        },
        "bulk" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "50"
        },
        "optimize" : {
          "type" : "fixed",
          "min" : 1,
          "max" : 1,
          "queue_size" : -1
        },
        "warmer" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "flush" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "search" : {
          "type" : "fixed",
          "min" : 96,
          "max" : 96,
          "queue_size" : "1k"
        },
        "percolate" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "1k"
        },
        "management" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "refresh" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 10,
          "keep_alive" : "5m",
          "queue_size" : -1
        }
      },
      "network" : {
        "refresh_interval_in_millis" : 5000,
        "primary_interface" : {
          "address" : "",
          "name" : "",
          "mac_address" : ""
        }
      },
      "transport" : {
        "bound_address" : "inet[/0:0:0:0:0:0:0:0%0:9302]",
        "publish_address" : "inet[/10.0.108.171:9302]"
      },
      "plugins" : [ ]
    },
    "myfhcxbYRCaqpJ_yAapmHw" : {
      "name" : "node38-telecom",
      "transport_address" : "inet[/10.0.108.171:9300]",
      "host" : "node38",
      "ip" : "10.0.108.171",
      "version" : "1.3.0",
      "build" : "1265b14",
      "http_address" : "inet[/10.0.108.171:9200]",
      "settings" : {
        "node" : {
          "name" : "node38-telecom"
        },
        "index" : {
          "number_of_replicas" : "1",
          "number_of_shards" : "8"
        },
        "bootstrap" : {
          "mlockall" : "true"
        },
        "name" : "node38-telecom",
        "pidfile" : "/var/run/elasticsearch/elasticsearch-telecom.pid",
        "path" : {
          "data" : "/var/lib/elasticsearch/telecom",
          "work" : "/tmp/elasticsearch",
          "home" : "/usr/share/elasticsearch",
          "conf" : "/etc/elasticsearch/telecom",
          "logs" : "/var/log/elasticsearch/telecom"
        },
        "cluster" : {
          "name" : "telecom"
        },
        "indices" : {
          "memory" : {
            "index_buffer_size" : "30%"
          }
        },
        "discovery" : {
          "zen" : {
            "minimum_master_nodes" : "1",
            "ping" : {
              "unicast" : {
                "hosts" : [ "node07", "node38" ]
              },
              "multicast" : {
                "enabled" : "false"
              },
              "timeout" : "30s"
            }
          }
        }
      },
      "os" : {
        "refresh_interval_in_millis" : 1000,
        "available_processors" : 32,
        "cpu" : {
          "vendor" : "Intel",
          "model" : "Xeon",
          "mhz" : 2000,
          "total_cores" : 32,
          "total_sockets" : 1,
          "cores_per_socket" : 32,
          "cache_size_in_bytes" : 20480
        },
        "mem" : {
          "total_in_bytes" : 33617100800
        },
        "swap" : {
          "total_in_bytes" : 17459511296
        }
      },
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 12731,
        "max_file_descriptors" : 65535,
        "mlockall" : false
      },
      "jvm" : {
        "pid" : 12731,
        "version" : "1.7.0_65",
        "vm_name" : "OpenJDK 64-Bit Server VM",
        "vm_version" : "24.65-b04",
        "vm_vendor" : "Oracle Corporation",
        "start_time_in_millis" : 1406268268972,
        "mem" : {
          "heap_init_in_bytes" : 17179869184,
          "heap_max_in_bytes" : 16979263488,
          "non_heap_init_in_bytes" : 24313856,
          "non_heap_max_in_bytes" : 224395264,
          "direct_max_in_bytes" : 16979263488
        },
        "gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
        "memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
      },
      "thread_pool" : {
        "generic" : {
          "type" : "cached",
          "keep_alive" : "30s",
          "queue_size" : -1
        },
        "index" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "200"
        },
        "snapshot_data" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "bench" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "get" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "1k"
        },
        "snapshot" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "merge" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "suggest" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "1k"
        },
        "bulk" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "50"
        },
        "optimize" : {
          "type" : "fixed",
          "min" : 1,
          "max" : 1,
          "queue_size" : -1
        },
        "warmer" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "flush" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "search" : {
          "type" : "fixed",
          "min" : 96,
          "max" : 96,
          "queue_size" : "1k"
        },
        "percolate" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "1k"
        },
        "management" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "refresh" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 10,
          "keep_alive" : "5m",
          "queue_size" : -1
        }
      },
      "network" : {
        "refresh_interval_in_millis" : 5000,
        "primary_interface" : {
          "address" : "10.0.108.171",
          "name" : "eth0",
          "mac_address" : ""
        }
      },
      "transport" : {
        "bound_address" : "inet[/0:0:0:0:0:0:0:0:9300]",
        "publish_address" : "inet[/10.0.108.171:9300]"
      },
      "http" : {
        "bound_address" : "inet[/0:0:0:0:0:0:0:0:9200]",
        "publish_address" : "inet[/10.0.108.171:9200]",
        "max_content_length_in_bytes" : 104857600
      },
      "plugins" : [ ]
    },
    "9FwiLpriR12926UVB-YuVw" : {
      "name" : "logstash-netflow",
      "transport_address" : "inet[/10.0.108.171:9301]",
      "host" : "node38",
      "ip" : "10.0.108.171",
      "version" : "1.1.1",
      "build" : "f1585f0",
      "attributes" : {
        "client" : "true",
        "data" : "false"
      },
      "settings" : {
        "path" : {
          "logs" : "/home/sysunix/logs"
        },
        "cluster" : {
          "name" : "telecom"
        },
        "node" : {
          "client" : "true",
          "name" : "logstash-netflow"
        },
        "discovery" : {
          "zen" : {
            "ping" : {
              "unicast" : {
                "hosts" : "node38:9300,node38:9301,node38:9302,node38:9303,node38:9304,node38:9305"
              },
              "multicast" : {
                "enabled" : "false"
              }
            }
          }
        },
        "http" : {
          "enabled" : "false"
        },
        "name" : "logstash-netflow"
      },
      "os" : {
        "refresh_interval_in_millis" : 1000,
        "available_processors" : 32
      },
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 8365,
        "max_file_descriptors" : 4096,
        "mlockall" : false
      },
      "jvm" : {
        "pid" : 8365,
        "version" : "1.7.0_55",
        "vm_name" : "OpenJDK 64-Bit Server VM",
        "vm_version" : "24.51-b03",
        "vm_vendor" : "Oracle Corporation",
        "start_time_in_millis" : 1402068863618,
        "mem" : {
          "heap_init_in_bytes" : 524288000,
          "heap_max_in_bytes" : 506855424,
          "non_heap_init_in_bytes" : 24313856,
          "non_heap_max_in_bytes" : 224395264,
          "direct_max_in_bytes" : 506855424
        },
        "gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
        "memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
      },
      "thread_pool" : {
        "generic" : {
          "type" : "cached",
          "keep_alive" : "30s",
          "queue_size" : -1
        },
        "index" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "200"
        },
        "get" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "1k"
        },
        "snapshot" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "merge" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "suggest" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "1k"
        },
        "bulk" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "50"
        },
        "optimize" : {
          "type" : "fixed",
          "min" : 1,
          "max" : 1,
          "queue_size" : -1
        },
        "warmer" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "flush" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "search" : {
          "type" : "fixed",
          "min" : 96,
          "max" : 96,
          "queue_size" : "1k"
        },
        "percolate" : {
          "type" : "fixed",
          "min" : 32,
          "max" : 32,
          "queue_size" : "1k"
        },
        "management" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "refresh" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 10,
          "keep_alive" : "5m",
          "queue_size" : -1
        }
      },
      "network" : {
        "refresh_interval_in_millis" : 5000,
        "primary_interface" : {
          "address" : "",
          "name" : "",
          "mac_address" : ""
        }
      },
      "transport" : {
        "bound_address" : "inet[/0:0:0:0:0:0:0:0%0:9301]",
        "publish_address" : "inet[/10.0.108.171:9301]"
      },
      "plugins" : [ ]
    }
  }
}
curl 0:9200/_nodes/stats

{
  "cluster_name" : "telecom",
  "nodes" : {
    "JUOZAr-mT6e8e9nmwWH9ww" : {
      "timestamp" : 1406277844538,
      "name" : "node07-telecom",
      "transport_address" : "inet[/10.0.104.214:9300]",
      "host" : "node07",
      "ip" : [ "inet[/10.0.104.214:9300]", "NONE" ],
      "indices" : {
        "docs" : {
          "count" : 0,
          "deleted" : 0
        },
        "store" : {
          "size_in_bytes" : 0,
          "throttle_time_in_millis" : 0
        },
        "indexing" : {
          "index_total" : 0,
          "index_time_in_millis" : 0,
          "index_current" : 0,
          "delete_total" : 0,
          "delete_time_in_millis" : 0,
          "delete_current" : 0
        },
        "get" : {
          "total" : 0,
          "time_in_millis" : 0,
          "exists_total" : 0,
          "exists_time_in_millis" : 0,
          "missing_total" : 0,
          "missing_time_in_millis" : 0,
          "current" : 0
        },
        "search" : {
          "open_contexts" : 0,
          "query_total" : 0,
          "query_time_in_millis" : 0,
          "query_current" : 0,
          "fetch_total" : 0,
          "fetch_time_in_millis" : 0,
          "fetch_current" : 0
        },
        "merges" : {
          "current" : 0,
          "current_docs" : 0,
          "current_size_in_bytes" : 0,
          "total" : 0,
          "total_time_in_millis" : 0,
          "total_docs" : 0,
          "total_size_in_bytes" : 0
        },
        "refresh" : {
          "total" : 0,
          "total_time_in_millis" : 0
        },
        "flush" : {
          "total" : 0,
          "total_time_in_millis" : 0
        },
        "warmer" : {
          "current" : 0,
          "total" : 0,
          "total_time_in_millis" : 0
        },
        "filter_cache" : {
          "memory_size_in_bytes" : 0,
          "evictions" : 0
        },
        "id_cache" : {
          "memory_size_in_bytes" : 0
        },
        "fielddata" : {
          "memory_size_in_bytes" : 0,
          "evictions" : 0
        },
        "percolate" : {
          "total" : 0,
          "time_in_millis" : 0,
          "current" : 0,
          "memory_size_in_bytes" : -1,
          "memory_size" : "-1b",
          "queries" : 0
        },
        "completion" : {
          "size_in_bytes" : 0
        },
        "segments" : {
          "count" : 0,
          "memory_in_bytes" : 0,
          "index_writer_memory_in_bytes" : 0,
          "version_map_memory_in_bytes" : 0
        },
        "translog" : {
          "operations" : 0,
          "size_in_bytes" : 0
        },
        "suggest" : {
          "total" : 0,
          "time_in_millis" : 0,
          "current" : 0
        }
      },
      "os" : {
        "timestamp" : 1406277844539,
        "uptime_in_millis" : 2050688,
        "load_average" : [ 0.0, 0.0, 0.0 ],
        "cpu" : {
          "sys" : 0,
          "user" : 0,
          "idle" : 99,
          "usage" : 0,
          "stolen" : 0
        },
        "mem" : {
          "free_in_bytes" : 13301526528,
          "used_in_bytes" : 11883552768,
          "free_percent" : 94,
          "used_percent" : 5,
          "actual_free_in_bytes" : 23725084672,
          "actual_used_in_bytes" : 1459994624
        },
        "swap" : {
          "used_in_bytes" : 26796032,
          "free_in_bytes" : 252854272
        }
      },
      "process" : {
        "timestamp" : 1406277844540,
        "open_file_descriptors" : 464,
        "cpu" : {
          "percent" : 0,
          "sys_in_millis" : 17990,
          "user_in_millis" : 33500,
          "total_in_millis" : 51490
        },
        "mem" : {
          "resident_in_bytes" : 616067072,
          "share_in_bytes" : 13942784,
          "total_virtual_in_bytes" : 26445426688
        }
      },
      "jvm" : {
        "timestamp" : 1406277844540,
        "uptime_in_millis" : 9502740,
        "mem" : {
          "heap_used_in_bytes" : 575853064,
          "heap_used_percent" : 3,
          "heap_committed_in_bytes" : 17066491904,
          "heap_max_in_bytes" : 17066491904,
          "non_heap_used_in_bytes" : 31055048,
          "non_heap_committed_in_bytes" : 32374784,
          "pools" : {
            "young" : {
              "used_in_bytes" : 546352176,
              "max_in_bytes" : 907345920,
              "peak_used_in_bytes" : 907345920,
              "peak_max_in_bytes" : 907345920
            },
            "survivor" : {
              "used_in_bytes" : 29500888,
              "max_in_bytes" : 113377280,
              "peak_used_in_bytes" : 29500888,
              "peak_max_in_bytes" : 113377280
            },
            "old" : {
              "used_in_bytes" : 0,
              "max_in_bytes" : 16045768704,
              "peak_used_in_bytes" : 0,
              "peak_max_in_bytes" : 16045768704
            }
          }
        },
        "threads" : {
          "count" : 112,
          "peak_count" : 115
        },
        "gc" : {
          "collectors" : {
            "young" : {
              "collection_count" : 1,
              "collection_time_in_millis" : 58
            },
            "old" : {
              "collection_count" : 0,
              "collection_time_in_millis" : 0
            }
          }
        },
        "buffer_pools" : {
          "direct" : {
            "count" : 179,
            "used_in_bytes" : 49807360,
            "total_capacity_in_bytes" : 49807360
          },
          "mapped" : {
            "count" : 0,
            "used_in_bytes" : 0,
            "total_capacity_in_bytes" : 0
          }
        }
      },
      "thread_pool" : {
        "generic" : {
          "threads" : 1,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 4,
          "completed" : 972
        },
        "index" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "snapshot_data" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "bench" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "get" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "snapshot" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "merge" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "suggest" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "bulk" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "optimize" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "warmer" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "flush" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "search" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "percolate" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "management" : {
          "threads" : 1,
          "queue" : 0,
          "active" : 1,
          "rejected" : 0,
          "largest" : 1,
          "completed" : 653
        },
        "refresh" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        }
      },
      "network" : {
        "tcp" : {
          "active_opens" : 421209,
          "passive_opens" : 138326,
          "curr_estab" : 132,
          "in_segs" : 110881241,
          "out_segs" : 114912697,
          "retrans_segs" : 105,
          "estab_resets" : 253951,
          "attempt_fails" : 4321,
          "in_errs" : 0,
          "out_rsts" : 68724493
        }
      },
      "fs" : {
        "timestamp" : 1406277844541,
        "total" : {
          "total_in_bytes" : 1648462135296,
          "free_in_bytes" : 1632006266880,
          "available_in_bytes" : 1632006266880,
          "disk_reads" : 11399,
          "disk_writes" : 3156978,
          "disk_io_op" : 3168377,
          "disk_read_size_in_bytes" : 515517952,
          "disk_write_size_in_bytes" : 302869040640,
          "disk_io_size_in_bytes" : 303384558592,
          "disk_queue" : "1.5E-4",
          "disk_service_time" : "0.1"
        },
        "data" : [ {
          "path" : "/var/lib/elasticsearch/telecom/telecom/nodes/0",
          "mount" : "/var/lib/elasticsearch",
          "dev" : "/dev/mapper/rootvg-elasticsearch",
          "total_in_bytes" : 1648462135296,
          "free_in_bytes" : 1632006266880,
          "available_in_bytes" : 1632006266880,
          "disk_reads" : 11399,
          "disk_writes" : 3156978,
          "disk_io_op" : 3168377,
          "disk_read_size_in_bytes" : 515517952,
          "disk_write_size_in_bytes" : 302869040640,
          "disk_io_size_in_bytes" : 303384558592,
          "disk_queue" : "1.5E-4",
          "disk_service_time" : "0.1"
        } ]
      },
      "transport" : {
        "server_open" : 65,
        "rx_count" : 19014,
        "rx_size_in_bytes" : 934110,
        "tx_count" : 19013,
        "tx_size_in_bytes" : 1151504
      },
      "http" : {
        "current_open" : 0,
        "total_opened" : 643
      },
      "fielddata_breaker" : {
        "maximum_size_in_bytes" : 10239895142,
        "maximum_size" : "9.5gb",
        "estimated_size_in_bytes" : 0,
        "estimated_size" : "0b",
        "overhead" : 1.03,
        "tripped" : 0
      }
    },
    "JL38-jS9Sn67hQk2XxNZxw" : {
      "timestamp" : 1406277844542,
      "name" : "logstash-netflow",
      "transport_address" : "inet[/10.0.108.171:9303]",
      "host" : "node38",
      "ip" : [ "inet[/10.0.108.171:9303]", "NONE" ],
      "attributes" : {
        "client" : "true",
        "data" : "false"
      },
      "indices" : {
        "docs" : {
          "count" : 0,
          "deleted" : 0
        },
        "store" : {
          "size_in_bytes" : 0,
          "throttle_time_in_millis" : 0
        },
        "indexing" : {
          "index_total" : 0,
          "index_time_in_millis" : 0,
          "index_current" : 0,
          "delete_total" : 0,
          "delete_time_in_millis" : 0,
          "delete_current" : 0
        },
        "get" : {
          "total" : 0,
          "time_in_millis" : 0,
          "exists_total" : 0,
          "exists_time_in_millis" : 0,
          "missing_total" : 0,
          "missing_time_in_millis" : 0,
          "current" : 0
        },
        "search" : {
          "open_contexts" : 0,
          "query_total" : 0,
          "query_time_in_millis" : 0,
          "query_current" : 0,
          "fetch_total" : 0,
          "fetch_time_in_millis" : 0,
          "fetch_current" : 0
        },
        "merges" : {
          "current" : 0,
          "current_docs" : 0,
          "current_size_in_bytes" : 0,
          "total" : 0,
          "total_time_in_millis" : 0,
          "total_docs" : 0,
          "total_size_in_bytes" : 0
        },
        "refresh" : {
          "total" : 0,
          "total_time_in_millis" : 0
        },
        "flush" : {
          "total" : 0,
          "total_time_in_millis" : 0
        },
        "warmer" : {
          "current" : 0,
          "total" : 0,
          "total_time_in_millis" : 0
        },
        "filter_cache" : {
          "memory_size_in_bytes" : 0,
          "evictions" : 0
        },
        "id_cache" : {
          "memory_size_in_bytes" : 0
        },
        "fielddata" : {
          "memory_size_in_bytes" : 0,
          "evictions" : 0
        },
        "percolate" : {
          "total" : 0,
          "time_in_millis" : 0,
          "current" : 0,
          "memory_size_in_bytes" : -1,
          "memory_size" : "-1b",
          "queries" : 0
        },
        "completion" : {
          "size_in_bytes" : 0
        },
        "segments" : {
          "count" : 0,
          "memory_in_bytes" : 0,
          "index_writer_memory_in_bytes" : 0,
          "version_map_memory_in_bytes" : 0
        },
        "translog" : {
          "operations" : 0,
          "size_in_bytes" : 0
        }
      },
      "os" : {
        "timestamp" : 1406277844542
      },
      "process" : {
        "timestamp" : 1406277844542,
        "open_file_descriptors" : 1377
      },
      "jvm" : {
        "timestamp" : 1406277844544,
        "uptime_in_millis" : 4208980926,
        "mem" : {
          "heap_used_in_bytes" : 226119440,
          "heap_used_percent" : 44,
          "heap_committed_in_bytes" : 506855424,
          "heap_max_in_bytes" : 506855424,
          "non_heap_used_in_bytes" : 68490160,
          "non_heap_committed_in_bytes" : 107167744,
          "pools" : {
            "young" : {
              "used_in_bytes" : 46953832,
              "max_in_bytes" : 139853824,
              "peak_used_in_bytes" : 139853824,
              "peak_max_in_bytes" : 139853824
            },
            "survivor" : {
              "used_in_bytes" : 110032,
              "max_in_bytes" : 17432576,
              "peak_used_in_bytes" : 17432576,
              "peak_max_in_bytes" : 17432576
            },
            "old" : {
              "used_in_bytes" : 179055576,
              "max_in_bytes" : 349569024,
              "peak_used_in_bytes" : 339622256,
              "peak_max_in_bytes" : 349569024
            }
          }
        },
        "threads" : {
          "count" : 428,
          "peak_count" : 442
        },
        "gc" : {
          "collectors" : {
            "young" : {
              "collection_count" : 11840,
              "collection_time_in_millis" : 67781
            },
            "old" : {
              "collection_count" : 14,
              "collection_time_in_millis" : 203
            }
          }
        },
        "buffer_pools" : {
          "direct" : {
            "count" : 723,
            "used_in_bytes" : 199419308,
            "total_capacity_in_bytes" : 199419308
          },
          "mapped" : {
            "count" : 0,
            "used_in_bytes" : 0,
            "total_capacity_in_bytes" : 0
          }
        }
      },
      "thread_pool" : {
        "generic" : {
          "threads" : 1,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 4,
          "completed" : 420988
        },
        "index" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "get" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "snapshot" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "merge" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "suggest" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "bulk" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "optimize" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "warmer" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "flush" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "search" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "percolate" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "management" : {
          "threads" : 1,
          "queue" : 0,
          "active" : 1,
          "rejected" : 0,
          "largest" : 2,
          "completed" : 766
        },
        "refresh" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        }
      },
      "network" : { },
      "fs" : {
        "timestamp" : 1406277844544,
        "total" : { },
        "data" : [ ]
      },
      "transport" : {
        "server_open" : 26,
        "rx_count" : 9160629,
        "rx_size_in_bytes" : 2570548643,
        "tx_count" : 9160591,
        "tx_size_in_bytes" : 1643358255
      },
      "fielddata_breaker" : {
        "maximum_size_in_bytes" : 405484339,
        "maximum_size" : "386.6mb",
        "estimated_size_in_bytes" : 0,
        "estimated_size" : "0b",
        "overhead" : 1.03,
        "tripped" : -1
      }
    },
    "rPZ9EsahRl-9Cs_AOucOJQ" : {
      "timestamp" : 1406277844542,
      "name" : "logstash-netflow",
      "transport_address" : "inet[/10.0.108.171:9302]",
      "host" : "node38",
      "ip" : [ "inet[/10.0.108.171:9302]", "NONE" ],
      "attributes" : {
        "client" : "true",
        "data" : "false"
      },
      "indices" : {
        "docs" : {
          "count" : 0,
          "deleted" : 0
        },
        "store" : {
          "size_in_bytes" : 0,
          "throttle_time_in_millis" : 0
        },
        "indexing" : {
          "index_total" : 0,
          "index_time_in_millis" : 0,
          "index_current" : 0,
          "delete_total" : 0,
          "delete_time_in_millis" : 0,
          "delete_current" : 0
        },
        "get" : {
          "total" : 0,
          "time_in_millis" : 0,
          "exists_total" : 0,
          "exists_time_in_millis" : 0,
          "missing_total" : 0,
          "missing_time_in_millis" : 0,
          "current" : 0
        },
        "search" : {
          "open_contexts" : 0,
          "query_total" : 0,
          "query_time_in_millis" : 0,
          "query_current" : 0,
          "fetch_total" : 0,
          "fetch_time_in_millis" : 0,
          "fetch_current" : 0
        },
        "merges" : {
          "current" : 0,
          "current_docs" : 0,
          "current_size_in_bytes" : 0,
          "total" : 0,
          "total_time_in_millis" : 0,
          "total_docs" : 0,
          "total_size_in_bytes" : 0
        },
        "refresh" : {
          "total" : 0,
          "total_time_in_millis" : 0
        },
        "flush" : {
          "total" : 0,
          "total_time_in_millis" : 0
        },
        "warmer" : {
          "current" : 0,
          "total" : 0,
          "total_time_in_millis" : 0
        },
        "filter_cache" : {
          "memory_size_in_bytes" : 0,
          "evictions" : 0
        },
        "id_cache" : {
          "memory_size_in_bytes" : 0
        },
        "fielddata" : {
          "memory_size_in_bytes" : 0,
          "evictions" : 0
        },
        "percolate" : {
          "total" : 0,
          "time_in_millis" : 0,
          "current" : 0,
          "memory_size_in_bytes" : -1,
          "memory_size" : "-1b",
          "queries" : 0
        },
        "completion" : {
          "size_in_bytes" : 0
        },
        "segments" : {
          "count" : 0,
          "memory_in_bytes" : 0,
          "index_writer_memory_in_bytes" : 0,
          "version_map_memory_in_bytes" : 0
        },
        "translog" : {
          "operations" : 0,
          "size_in_bytes" : 0
        }
      },
      "os" : {
        "timestamp" : 1406277844543
      },
      "process" : {
        "timestamp" : 1406277844543,
        "open_file_descriptors" : 1375
      },
      "jvm" : {
        "timestamp" : 1406277844544,
        "uptime_in_millis" : 4208980926,
        "mem" : {
          "heap_used_in_bytes" : 226123632,
          "heap_used_percent" : 44,
          "heap_committed_in_bytes" : 506855424,
          "heap_max_in_bytes" : 506855424,
          "non_heap_used_in_bytes" : 68490160,
          "non_heap_committed_in_bytes" : 107167744,
          "pools" : {
            "young" : {
              "used_in_bytes" : 46960112,
              "max_in_bytes" : 139853824,
              "peak_used_in_bytes" : 139853824,
              "peak_max_in_bytes" : 139853824
            },
            "survivor" : {
              "used_in_bytes" : 110032,
              "max_in_bytes" : 17432576,
              "peak_used_in_bytes" : 17432576,
              "peak_max_in_bytes" : 17432576
            },
            "old" : {
              "used_in_bytes" : 179055576,
              "max_in_bytes" : 349569024,
              "peak_used_in_bytes" : 339622256,
              "peak_max_in_bytes" : 349569024
            }
          }
        },
        "threads" : {
          "count" : 428,
          "peak_count" : 442
        },
        "gc" : {
          "collectors" : {
            "young" : {
              "collection_count" : 11840,
              "collection_time_in_millis" : 67781
            },
            "old" : {
              "collection_count" : 14,
              "collection_time_in_millis" : 203
            }
          }
        },
        "buffer_pools" : {
          "direct" : {
            "count" : 723,
            "used_in_bytes" : 199419308,
            "total_capacity_in_bytes" : 199419308
          },
          "mapped" : {
            "count" : 0,
            "used_in_bytes" : 0,
            "total_capacity_in_bytes" : 0
          }
        }
      },
      "thread_pool" : {
        "generic" : {
          "threads" : 1,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 3,
          "completed" : 420986
        },
        "index" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "get" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "snapshot" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "merge" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "suggest" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "bulk" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "optimize" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "warmer" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "flush" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "search" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "percolate" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "management" : {
          "threads" : 1,
          "queue" : 0,
          "active" : 1,
          "rejected" : 0,
          "largest" : 2,
          "completed" : 766
        },
        "refresh" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        }
      },
      "network" : { },
      "fs" : {
        "timestamp" : 1406277844544,
        "total" : { },
        "data" : [ ]
      },
      "transport" : {
        "server_open" : 26,
        "rx_count" : 9160652,
        "rx_size_in_bytes" : 2567605017,
        "tx_count" : 9160605,
        "tx_size_in_bytes" : 1635812229
      },
      "fielddata_breaker" : {
        "maximum_size_in_bytes" : 405484339,
        "maximum_size" : "386.6mb",
        "estimated_size_in_bytes" : 0,
        "estimated_size" : "0b",
        "overhead" : 1.03,
        "tripped" : -1
      }
    },
    "myfhcxbYRCaqpJ_yAapmHw" : {
      "timestamp" : 1406277844542,
      "name" : "node38-telecom",
      "transport_address" : "inet[/10.0.108.171:9300]",
      "host" : "node38",
      "ip" : [ "inet[/10.0.108.171:9300]", "NONE" ],
      "indices" : {
        "docs" : {
          "count" : 33434392,
          "deleted" : 0
        },
        "store" : {
          "size_in_bytes" : 16439595838,
          "throttle_time_in_millis" : 0
        },
        "indexing" : {
          "index_total" : 0,
          "index_time_in_millis" : 0,
          "index_current" : 0,
          "delete_total" : 0,
          "delete_time_in_millis" : 0,
          "delete_current" : 0
        },
        "get" : {
          "total" : 0,
          "time_in_millis" : 0,
          "exists_total" : 0,
          "exists_time_in_millis" : 0,
          "missing_total" : 0,
          "missing_time_in_millis" : 0,
          "current" : 0
        },
        "search" : {
          "open_contexts" : 0,
          "query_total" : 0,
          "query_time_in_millis" : 0,
          "query_current" : 0,
          "fetch_total" : 0,
          "fetch_time_in_millis" : 0,
          "fetch_current" : 0
        },
        "merges" : {
          "current" : 0,
          "current_docs" : 0,
          "current_size_in_bytes" : 0,
          "total" : 0,
          "total_time_in_millis" : 0,
          "total_docs" : 0,
          "total_size_in_bytes" : 0
        },
        "refresh" : {
          "total" : 26,
          "total_time_in_millis" : 0
        },
        "flush" : {
          "total" : 0,
          "total_time_in_millis" : 0
        },
        "warmer" : {
          "current" : 0,
          "total" : 52,
          "total_time_in_millis" : 2
        },
        "filter_cache" : {
          "memory_size_in_bytes" : 0,
          "evictions" : 0
        },
        "id_cache" : {
          "memory_size_in_bytes" : 0
        },
        "fielddata" : {
          "memory_size_in_bytes" : 0,
          "evictions" : 0
        },
        "percolate" : {
          "total" : 0,
          "time_in_millis" : 0,
          "current" : 0,
          "memory_size_in_bytes" : -1,
          "memory_size" : "-1b",
          "queries" : 0
        },
        "completion" : {
          "size_in_bytes" : 0
        },
        "segments" : {
          "count" : 331,
          "memory_in_bytes" : 380822712,
          "index_writer_memory_in_bytes" : 0,
          "version_map_memory_in_bytes" : 0
        },
        "translog" : {
          "operations" : 0,
          "size_in_bytes" : 0
        },
        "suggest" : {
          "total" : 0,
          "time_in_millis" : 0,
          "current" : 0
        }
      },
      "os" : {
        "timestamp" : 1406277844605,
        "uptime_in_millis" : 11125608,
        "load_average" : [ 0.02, 0.02, 0.0 ],
        "cpu" : {
          "sys" : 0,
          "user" : 0,
          "idle" : 99,
          "usage" : 0,
          "stolen" : 0
        },
        "mem" : {
          "free_in_bytes" : 9554894848,
          "used_in_bytes" : 24062205952,
          "free_percent" : 85,
          "used_percent" : 14,
          "actual_free_in_bytes" : 28848246784,
          "actual_used_in_bytes" : 4768854016
        },
        "swap" : {
          "used_in_bytes" : 306176000,
          "free_in_bytes" : 17153335296
        }
      },
      "process" : {
        "timestamp" : 1406277844605,
        "open_file_descriptors" : 1796,
        "cpu" : {
          "percent" : 1,
          "sys_in_millis" : 64390,
          "user_in_millis" : 113440,
          "total_in_millis" : 177830
        },
        "mem" : {
          "resident_in_bytes" : 2292084736,
          "share_in_bytes" : 16846848,
          "total_virtual_in_bytes" : 38485655552
        }
      },
      "jvm" : {
        "timestamp" : 1406277844607,
        "uptime_in_millis" : 9575635,
        "mem" : {
          "heap_used_in_bytes" : 1142720912,
          "heap_used_percent" : 6,
          "heap_committed_in_bytes" : 16979263488,
          "heap_max_in_bytes" : 16979263488,
          "non_heap_used_in_bytes" : 39112624,
          "non_heap_committed_in_bytes" : 39387136,
          "pools" : {
            "young" : {
              "used_in_bytes" : 978794664,
              "max_in_bytes" : 1605304320,
              "peak_used_in_bytes" : 1605304320,
              "peak_max_in_bytes" : 1605304320
            },
            "survivor" : {
              "used_in_bytes" : 3106256,
              "max_in_bytes" : 200605696,
              "peak_used_in_bytes" : 166923624,
              "peak_max_in_bytes" : 200605696
            },
            "old" : {
              "used_in_bytes" : 160819992,
              "max_in_bytes" : 15173353472,
              "peak_used_in_bytes" : 160819992,
              "peak_max_in_bytes" : 15173353472
            }
          }
        },
        "threads" : {
          "count" : 214,
          "peak_count" : 229
        },
        "gc" : {
          "collectors" : {
            "young" : {
              "collection_count" : 7,
              "collection_time_in_millis" : 403
            },
            "old" : {
              "collection_count" : 0,
              "collection_time_in_millis" : 0
            }
          }
        },
        "buffer_pools" : {
          "direct" : {
            "count" : 336,
            "used_in_bytes" : 87049216,
            "total_capacity_in_bytes" : 87049216
          },
          "mapped" : {
            "count" : 440,
            "used_in_bytes" : 3388615670,
            "total_capacity_in_bytes" : 3388615670
          }
        }
      },
      "thread_pool" : {
        "generic" : {
          "threads" : 1,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 19,
          "completed" : 1054
        },
        "index" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "snapshot_data" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "bench" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "get" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "snapshot" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "merge" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "suggest" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "bulk" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "optimize" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "warmer" : {
          "threads" : 1,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 1,
          "completed" : 26
        },
        "flush" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "search" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "percolate" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "management" : {
          "threads" : 5,
          "queue" : 0,
          "active" : 1,
          "rejected" : 0,
          "largest" : 5,
          "completed" : 1354
        },
        "refresh" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        }
      },
      "network" : {
        "tcp" : {
          "active_opens" : 140097716,
          "passive_opens" : 1880178,
          "curr_estab" : 290,
          "in_segs" : 5172144515,
          "out_segs" : 5814393262,
          "retrans_segs" : 212947,
          "estab_resets" : 1056639,
          "attempt_fails" : 27282,
          "in_errs" : 50905,
          "out_rsts" : 1112514
        }
      },
      "fs" : {
        "timestamp" : 1406277844608,
        "total" : {
          "total_in_bytes" : 2197949513728,
          "free_in_bytes" : 2181463736320,
          "available_in_bytes" : 2181463736320,
          "disk_reads" : 38753157,
          "disk_writes" : 243000636,
          "disk_io_op" : 281753793,
          "disk_read_size_in_bytes" : 4230850646528,
          "disk_write_size_in_bytes" : 13743848959488,
          "disk_io_size_in_bytes" : 17974699606016
        },
        "data" : [ {
          "path" : "/var/lib/elasticsearch/telecom/telecom/nodes/0",
          "mount" : "/var/lib/elasticsearch",
          "dev" : "/dev/mapper/raidvg-elasticsearch",
          "total_in_bytes" : 2197949513728,
          "free_in_bytes" : 2181463736320,
          "available_in_bytes" : 2181463736320,
          "disk_reads" : 38753157,
          "disk_writes" : 243000636,
          "disk_io_op" : 281753793,
          "disk_read_size_in_bytes" : 4230850646528,
          "disk_write_size_in_bytes" : 13743848959488,
          "disk_io_size_in_bytes" : 17974699606016
        } ]
      },
      "transport" : {
        "server_open" : 65,
        "rx_count" : 76220,
        "rx_size_in_bytes" : 4624394,
        "tx_count" : 76219,
        "tx_size_in_bytes" : 3602292
      },
      "http" : {
        "current_open" : 1,
        "total_opened" : 655
      },
      "fielddata_breaker" : {
        "maximum_size_in_bytes" : 10187558092,
        "maximum_size" : "9.4gb",
        "estimated_size_in_bytes" : 0,
        "estimated_size" : "0b",
        "overhead" : 1.03,
        "tripped" : 0
      }
    },
    "9FwiLpriR12926UVB-YuVw" : {
      "timestamp" : 1406277844542,
      "name" : "logstash-netflow",
      "transport_address" : "inet[/10.0.108.171:9301]",
      "host" : "node38",
      "ip" : [ "inet[/10.0.108.171:9301]", "NONE" ],
      "attributes" : {
        "client" : "true",
        "data" : "false"
      },
      "indices" : {
        "docs" : {
          "count" : 0,
          "deleted" : 0
        },
        "store" : {
          "size_in_bytes" : 0,
          "throttle_time_in_millis" : 0
        },
        "indexing" : {
          "index_total" : 0,
          "index_time_in_millis" : 0,
          "index_current" : 0,
          "delete_total" : 0,
          "delete_time_in_millis" : 0,
          "delete_current" : 0
        },
        "get" : {
          "total" : 0,
          "time_in_millis" : 0,
          "exists_total" : 0,
          "exists_time_in_millis" : 0,
          "missing_total" : 0,
          "missing_time_in_millis" : 0,
          "current" : 0
        },
        "search" : {
          "open_contexts" : 0,
          "query_total" : 0,
          "query_time_in_millis" : 0,
          "query_current" : 0,
          "fetch_total" : 0,
          "fetch_time_in_millis" : 0,
          "fetch_current" : 0
        },
        "merges" : {
          "current" : 0,
          "current_docs" : 0,
          "current_size_in_bytes" : 0,
          "total" : 0,
          "total_time_in_millis" : 0,
          "total_docs" : 0,
          "total_size_in_bytes" : 0
        },
        "refresh" : {
          "total" : 0,
          "total_time_in_millis" : 0
        },
        "flush" : {
          "total" : 0,
          "total_time_in_millis" : 0
        },
        "warmer" : {
          "current" : 0,
          "total" : 0,
          "total_time_in_millis" : 0
        },
        "filter_cache" : {
          "memory_size_in_bytes" : 0,
          "evictions" : 0
        },
        "id_cache" : {
          "memory_size_in_bytes" : 0
        },
        "fielddata" : {
          "memory_size_in_bytes" : 0,
          "evictions" : 0
        },
        "percolate" : {
          "total" : 0,
          "time_in_millis" : 0,
          "current" : 0,
          "memory_size_in_bytes" : -1,
          "memory_size" : "-1b",
          "queries" : 0
        },
        "completion" : {
          "size_in_bytes" : 0
        },
        "segments" : {
          "count" : 0,
          "memory_in_bytes" : 0,
          "index_writer_memory_in_bytes" : 0,
          "version_map_memory_in_bytes" : 0
        },
        "translog" : {
          "operations" : 0,
          "size_in_bytes" : 0
        }
      },
      "os" : {
        "timestamp" : 1406277844543
      },
      "process" : {
        "timestamp" : 1406277844543,
        "open_file_descriptors" : 1376
      },
      "jvm" : {
        "timestamp" : 1406277844544,
        "uptime_in_millis" : 4208980926,
        "mem" : {
          "heap_used_in_bytes" : 226119440,
          "heap_used_percent" : 44,
          "heap_committed_in_bytes" : 506855424,
          "heap_max_in_bytes" : 506855424,
          "non_heap_used_in_bytes" : 68490160,
          "non_heap_committed_in_bytes" : 107167744,
          "pools" : {
            "young" : {
              "used_in_bytes" : 46955928,
              "max_in_bytes" : 139853824,
              "peak_used_in_bytes" : 139853824,
              "peak_max_in_bytes" : 139853824
            },
            "survivor" : {
              "used_in_bytes" : 110032,
              "max_in_bytes" : 17432576,
              "peak_used_in_bytes" : 17432576,
              "peak_max_in_bytes" : 17432576
            },
            "old" : {
              "used_in_bytes" : 179055576,
              "max_in_bytes" : 349569024,
              "peak_used_in_bytes" : 339622256,
              "peak_max_in_bytes" : 349569024
            }
          }
        },
        "threads" : {
          "count" : 428,
          "peak_count" : 442
        },
        "gc" : {
          "collectors" : {
            "young" : {
              "collection_count" : 11840,
              "collection_time_in_millis" : 67781
            },
            "old" : {
              "collection_count" : 14,
              "collection_time_in_millis" : 203
            }
          }
        },
        "buffer_pools" : {
          "direct" : {
            "count" : 723,
            "used_in_bytes" : 199419308,
            "total_capacity_in_bytes" : 199419308
          },
          "mapped" : {
            "count" : 0,
            "used_in_bytes" : 0,
            "total_capacity_in_bytes" : 0
          }
        }
      },
      "thread_pool" : {
        "generic" : {
          "threads" : 1,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 4,
          "completed" : 420979
        },
        "index" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "get" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "snapshot" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "merge" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "suggest" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "bulk" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "optimize" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "warmer" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "flush" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "search" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "percolate" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        },
        "management" : {
          "threads" : 1,
          "queue" : 0,
          "active" : 1,
          "rejected" : 0,
          "largest" : 2,
          "completed" : 766
        },
        "refresh" : {
          "threads" : 0,
          "queue" : 0,
          "active" : 0,
          "rejected" : 0,
          "largest" : 0,
          "completed" : 0
        }
      },
      "network" : { },
      "fs" : {
        "timestamp" : 1406277844544,
        "total" : { },
        "data" : [ ]
      },
      "transport" : {
        "server_open" : 26,
        "rx_count" : 8688419,
        "rx_size_in_bytes" : 2179836154,
        "tx_count" : 8688418,
        "tx_size_in_bytes" : 486036255
      },
      "fielddata_breaker" : {
        "maximum_size_in_bytes" : 405484339,
        "maximum_size" : "386.6mb",
        "estimated_size_in_bytes" : 0,
        "estimated_size" : "0b",
        "overhead" : 1.03,
        "tripped" : -1
      }
    }
  }
}

@clintongormley
Copy link

thanks @faxm0dem

@drewr could you take another look at this with the new info that has been provided please?

@drewr
Copy link
Contributor

drewr commented Aug 19, 2014

Sorry, just now seeing this... @faxm0dem did anything else anomalous happen in the cluster before this? Like, nodes not communicating properly, or running out of mem, anything like that?

@faxm0dem
Copy link

Not really. I just upgraded my prod cluster to 1.3.2, and have the same issue.

@clintongormley clintongormley added the help wanted adoptme label Sep 8, 2014
@chandanbansal
Copy link

Getting same error after connecting with logstash. Its working fine as single node and in cluster mode without logstash. Tried with ES 1.3.2 and 1.24 with logstash 1.4.2. is there any workaround for same? But data indexing and query is working fine.

@faxm0dem
Copy link

I don't have logstash nodes in my devel cluster, and I still get the error

@imotov imotov mentioned this issue Sep 22, 2014
Merged
imotov added a commit to imotov/elasticsearch that referenced this issue Sep 23, 2014
@imotov
Fix NPE in /_cat/nodes
8b67107 
Query Cache and Suggest stats were introduced in v1.4.0_beta1 and 1.2.0 respectively, therefore they can be null while stats are received from older nodes in the cluster.

Fixes elastic#6297
imotov added a commit that referenced this issue Sep 23, 2014
@imotov
Fix NPE in /_cat/nodes
c8d9934 
Query Cache and Suggest stats were introduced in v1.4.0_beta1 and 1.2.0 respectively, therefore they can be null while stats are received from older nodes in the cluster.

Fixes #6297
@clintongormley clintongormley removed the help wanted adoptme label Sep 24, 2014
@imotov
Copy link
Contributor

imotov commented Sep 27, 2014

Fixed by #7815.

@imotov imotov closed this as completed Sep 27, 2014
@mcnewton
Copy link

Just to confirm - was still seeing this in 1.3.4, but upgraded to 1.4.0 today and it is now working correctly.
Thanks!
Matthew

@faxm0dem
Copy link

I confirm this is finally working!

@Grauen
Copy link

Grauen commented Nov 19, 2014

I upgraded to 1.4.0 today and now I see this error.

http://localhost:9200/_aliases?pretty=1
{
"error" : "RemoteTransportException[[Raza][inet[/123.456.17.4:9300]][indices:admin/get]]; nested: ActionNotFoundTransportException[No handler for action [indices:admin/get]]; ",
"status" : 500
}

http://0:9200/_cat/indices
{"error":"NullPointerException[null]","status":500}

However I am able to search indices directly

http://localhost:9200/testindex/_search?
{"took":2,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}

@faxm0dem
Copy link

@Grauen are all your nodes 1.4.0?

@clintongormley
Copy link

@Grauen This sounds like a bug that was fixed in #8387. if you are still seeing this problems when all of your nodes are running 1.4.0, then please open a new issue.

drewr added a commit to drewr/elasticsearch that referenced this issue Feb 28, 2015
@drewr
_cat/nodes: Thread null handling through stats and info
c7e638a 
Closes elastic#6297
@drewr drewr mentioned this issue Mar 1, 2015
Merged
drewr added a commit that referenced this issue Mar 18, 2015
@drewr
_cat/nodes: Thread null handling through stats and info
681b261 
Closes #6297

# Conflicts:
#	src/main/java/org/elasticsearch/rest/action/cat/RestNodesAction.java
drewr added a commit that referenced this issue Mar 18, 2015
@drewr
_cat/nodes: Thread null handling through stats and info
df647d5 
Closes #6297

# Conflicts:
#	src/main/java/org/elasticsearch/rest/action/cat/RestNodesAction.java
mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015
@drewr
_cat/nodes: Thread null handling through stats and info
e91cd24 
Closes elastic#6297

# Conflicts:
#	src/main/java/org/elasticsearch/rest/action/cat/RestNodesAction.java
mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015
@imotov
Fix NPE in /_cat/nodes
e29c512 
Query Cache and Suggest stats were introduced in v1.4.0_beta1 and 1.2.0 respectively, therefore they can be null while stats are received from older nodes in the cluster.

Fixes elastic#6297
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@imotov imotov

Labels
>bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests