{ls} provides monitoring APIs for retrieving runtime metrics about {ls}:
You can use the root resource to retrieve general information about the Logstash instance, including the host and version.
curl -XGET 'localhost:9600/?pretty'Example response:
{
"host": "skywalker",
"version": "{logstash_version}",
"http_address": "127.0.0.1:9600"
}|
Note
|
By default, the monitoring API attempts to bind to tcp:9600. If this port is already in use by another Logstash
instance, you need to launch Logstash with the --api.http.port flag specified to bind to a different port. See
[command-line-flags] for more information.
|
The {ls} Monitoring APIs are not secured by default, but you can configure {ls} to secure them in one of several ways to meet your organization’s needs.
You can enable SSL for the Logstash API by setting api.ssl.enabled: true in the logstash.yml, and providing the relevant keystore settings api.ssl.keystore.path and api.ssl.keystore.password:
api.ssl.enabled: true
api.ssl.keystore.path: /path/to/keystore.jks
api.ssl.keystore.password: "s3cUr3p4$$w0rd"The keystore must be in either jks or p12 format, and must contain both a certificate and a private key. Connecting clients receive this certificate, allowing them to authenticate the Logstash endpoint.
You can also require HTTP Basic authentication by setting api.auth.type: basic in the logstash.yml, and providing the relevant credentials api.auth.basic.username and api.auth.basic.password:
api.auth.type: basic
api.auth.basic.username: "logstash"
api.auth.basic.password: "s3cUreP4$$w0rD"|
Note
|
Usage of Keystore or Environment or variable replacements is encouraged for password-type fields to avoid storing them in plain text.
For example, specifying the value "${HTTP_PASS}" will resolve to the value stored in the secure keystore’s HTTP_PASS variable if present or the same variable from the environment)
|
The following options can be applied to all of the Logstash monitoring APIs.
When appending ?pretty=true to any request made, the JSON returned
will be pretty formatted (use it for debugging only!).
|
Note
|
For Logstash {logstash_version}, the human option is supported for the Hot Threads API
only. When you specify human=true, the results are returned in plain text instead of
JSON format. The default is false.
|
Statistics are returned in a format suitable for humans
(eg "exists_time": "1h" or "size": "1kb") and for computers
(eg "exists_time_in_millis": 3600000 or "size_in_bytes": 1024).
The human-readable values can be turned off by adding ?human=false
to the query string. This makes sense when the stats results are
being consumed by a monitoring tool, rather than intended for human
consumption. The default for the human flag is
false.
The node info API retrieves information about the node.
curl -XGET 'localhost:9600/_node/<types>'Where <types> is optional and specifies the types of node info you want to return.
You can limit the info that’s returned by combining any of the following types in a comma-separated list:
pipelines
|
Gets pipeline-specific information and settings for each pipeline. |
os
|
Gets node-level info about the OS. |
jvm
|
Gets node-level JVM info, including info about threads. |
See Common Options for a list of options that can be applied to all Logstash monitoring APIs.
The following request returns a JSON document that shows pipeline info, such as the number of workers, batch size, and batch delay:
curl -XGET 'localhost:9600/_node/pipelines?pretty'If you want to view additional information about a pipeline, such as stats for each configured input, filter, or output stage, see the Pipeline stats section under the Node Stats API.
Example response:
{
"pipelines" : {
"test" : {
"workers" : 1,
"batch_size" : 1,
"batch_delay" : 5,
"config_reload_automatic" : false,
"config_reload_interval" : 3
},
"test2" : {
"workers" : 8,
"batch_size" : 125,
"batch_delay" : 5,
"config_reload_automatic" : false,
"config_reload_interval" : 3
}
}You can see the info for a specific pipeline by including the pipeline ID. In
the following example, the ID of the pipeline is test:
curl -XGET 'localhost:9600/_node/pipelines/test?pretty'Example response:
{
"pipelines" : {
"test" : {
"workers" : 1,
"batch_size" : 1,
"batch_delay" : 5,
"config_reload_automatic" : false,
"config_reload_interval" : 3
}
}If you specify an invalid pipeline ID, the request returns a 404 Not Found error.
The following request returns a JSON document that shows the OS name, architecture, version, and available processors:
curl -XGET 'localhost:9600/_node/os?pretty'Example response:
{
"os": {
"name": "Mac OS X",
"arch": "x86_64",
"version": "10.12.4",
"available_processors": 8
}The following request returns a JSON document that shows node-level JVM stats, such as the JVM process id, version, VM info, memory usage, and info about garbage collectors:
curl -XGET 'localhost:9600/_node/jvm?pretty'Example response:
{
"jvm": {
"pid": 59616,
"version": "1.8.0_65",
"vm_name": "Java HotSpot(TM) 64-Bit Server VM",
"vm_version": "1.8.0_65",
"vm_vendor": "Oracle Corporation",
"start_time_in_millis": 1484251185878,
"mem": {
"heap_init_in_bytes": 268435456,
"heap_max_in_bytes": 1037959168,
"non_heap_init_in_bytes": 2555904,
"non_heap_max_in_bytes": 0
},
"gc_collectors": [
"ParNew",
"ConcurrentMarkSweep"
]
}
}The plugins info API gets information about all Logstash plugins that are currently installed.
This API basically returns the output of running the bin/logstash-plugin list --verbose command.
curl -XGET 'localhost:9600/_node/plugins?pretty'See Common Options for a list of options that can be applied to all Logstash monitoring APIs.
The output is a JSON document.
Example response:
{
"total": 93,
"plugins": [
{
"name": "logstash-codec-cef",
"version": "4.1.2"
},
{
"name": "logstash-codec-collectd",
"version": "3.0.3"
},
{
"name": "logstash-codec-dots",
"version": "3.0.2"
},
{
"name": "logstash-codec-edn",
"version": "3.0.2"
},
.
.
.
]The node stats API retrieves runtime stats about Logstash.
curl -XGET 'localhost:9600/_node/stats/<types>'Where <types> is optional and specifies the types of stats you want to return.
By default, all stats are returned. You can limit the info that’s returned by combining any of the following types in a comma-separated list:
jvm
|
Gets JVM stats, including stats about threads, memory usage, garbage collectors, and uptime. |
process
|
Gets process stats, including stats about file descriptors, memory consumption, and CPU usage. |
events
|
Gets event-related statistics for the Logstash instance (regardless of how many pipelines were created and destroyed). |
flow
|
Gets flow-related statistics for the Logstash instance (regardless of how many pipelines were created and destroyed). |
pipelines
|
Gets runtime stats about each Logstash pipeline. |
reloads
|
Gets runtime stats about config reload successes and failures. |
os
|
Gets runtime stats about cgroups when Logstash is running in a container. |
geoip_download_manager
|
Gets stats for databases used with the Geoip filter plugin. |
See Common Options for a list of options that can be applied to all Logstash monitoring APIs.
The following request returns a JSON document containing JVM stats:
curl -XGET 'localhost:9600/_node/stats/jvm?pretty'Example response:
{
"jvm" : {
"threads" : {
"count" : 49,
"peak_count" : 50
},
"mem" : {
"heap_used_percent" : 14,
"heap_committed_in_bytes" : 309866496,
"heap_max_in_bytes" : 1037959168,
"heap_used_in_bytes" : 151686096,
"non_heap_used_in_bytes" : 122486176,
"non_heap_committed_in_bytes" : 133222400,
"pools" : {
"survivor" : {
"peak_used_in_bytes" : 8912896,
"used_in_bytes" : 288776,
"peak_max_in_bytes" : 35782656,
"max_in_bytes" : 35782656,
"committed_in_bytes" : 8912896
},
"old" : {
"peak_used_in_bytes" : 148656848,
"used_in_bytes" : 148656848,
"peak_max_in_bytes" : 715849728,
"max_in_bytes" : 715849728,
"committed_in_bytes" : 229322752
},
"young" : {
"peak_used_in_bytes" : 71630848,
"used_in_bytes" : 2740472,
"peak_max_in_bytes" : 286326784,
"max_in_bytes" : 286326784,
"committed_in_bytes" : 71630848
}
}
},
"gc" : {
"collectors" : {
"old" : {
"collection_time_in_millis" : 607,
"collection_count" : 12
},
"young" : {
"collection_time_in_millis" : 4904,
"collection_count" : 1033
}
}
},
"uptime_in_millis" : 1809643
}The following request returns a JSON document containing process stats:
curl -XGET 'localhost:9600/_node/stats/process?pretty'Example response:
{
"process" : {
"open_file_descriptors" : 184,
"peak_open_file_descriptors" : 185,
"max_file_descriptors" : 10240,
"mem" : {
"total_virtual_in_bytes" : 5486125056
},
"cpu" : {
"total_in_millis" : 657136,
"percent" : 2,
"load_average" : {
"1m" : 2.38134765625
}
}
}The following request returns a JSON document containing event-related statistics for the Logstash instance:
curl -XGET 'localhost:9600/_node/stats/events?pretty'Example response:
{
"events" : {
"in" : 293658,
"filtered" : 293658,
"out" : 293658,
"duration_in_millis" : 2324391,
"queue_push_duration_in_millis" : 343816
}The following request returns a JSON document containing flow-rates for the Logstash instance:
curl -XGET 'localhost:9600/_node/stats/flow?pretty'Example response:
{
"flow" : {
"input_throughput" : {
"current": 189.720,
"lifetime": 201.841
},
"filter_throughput" : {
"current": 187.810,
"lifetime": 201.799
},
"output_throughput" : {
"current": 191.087,
"lifetime": 201.761
},
"queue_backpressure" : {
"current": 0.277,
"lifetime": 0.031
},
"worker_concurrency" : {
"current": 1.973,
"lifetime": 1.721
}
}
}Flow rates provide visibility into how a Logstash instance or an individual pipeline is currently performing relative to itself over time. This allows us to attach meaning to the cumulative-value metrics that are also presented by this API, and to determine whether an instance or pipeline is behaving better or worse than it has in the past.
| Flow Rate | Definition |
|---|---|
|
This metric is expressed in events-per-second, and is the rate of events being pushed into the pipeline(s) queue(s) relative to wall-clock time ( |
|
This metric is expressed in events-per-second, and is the rate of events flowing through the filter phase of the pipeline(s) relative to wall-clock time ( |
|
This metric is expressed in events-per-second, and is the rate of events flowing through the output phase of the pipeline(s) relative to wall-clock time ( |
|
This is a unitless metric representing the cumulative time spent by all workers relative to wall-clock time ( A pipeline is considered "saturated" when its A process is also considered "saturated" when its top-level |
|
This is a unitless metric representing the cumulative time spent by all inputs blocked pushing events into their pipeline’s queue, relative to wall-clock time ( While a "zero" value indicates no back-pressure to the queue, the magnitude of this metric is highly dependent on the shape of the pipelines and their inputs. It cannot be used to compare one pipeline to another or even one process to itself if the quantity or shape of its pipelines changes. A pipeline with only one single-threaded input may contribute up to 1.00, a pipeline whose inputs have hundreds of inbound connections may contribute much higher numbers to this combined value. Additionally, some amount of back-pressure is both normal and expected for pipelines that are pulling data, as this back-pressure allows them to slow down and pull data at a rate its downstream pipeline can tolerate. |
|
Only available when pipeline is using persisted queue.
This metric is the growth rate of bytes being pushed into the pipeline’s persisted queue relative to wall-clock time ( For the ideal condition, this metric should trend toward zero (or negative). |
|
Only available when pipeline is using persisted queue.
This metric is the growth rate of events being pushed into the pipeline’s persisted queue relative to wall-clock time ( For the ideal condition, this metric should trend toward zero (or negative). |
Each flow stat includes rates for one or more recent windows of time:
| Flow Window | Availability | Definition |
|---|---|---|
|
Stable |
the most recent ~10s |
|
Stable |
the lifetime of the relevant pipeline or process |
|
Technology Preview |
the most recent ~1 minute |
|
Technology Preview |
the most recent ~5 minutes |
|
Technology Preview |
the most recent ~15 minutes |
|
Technology Preview |
the most recent ~1 hour |
|
Technology Preview |
the most recent ~24 hours |
|
Note
|
The flow rate windows marked as "Technology Preview" are subject to change without notice. Future releases of {ls} may include more, fewer, or different windows for each rate in response to community feedback. |
The following request returns a JSON document containing pipeline stats, including:
-
the number of events that were input, filtered, or output by each pipeline
-
the current and lifetime flow rates for each pipeline
-
stats for each configured filter or output stage
-
info about config reload successes and failures (when config reload is enabled)
-
info about the persistent queue (when persistent queues are enabled)
curl -XGET 'localhost:9600/_node/stats/pipelines?pretty'Example response:
{
"pipelines" : {
"test" : {
"events" : {
"duration_in_millis" : 365495,
"in" : 216485,
"filtered" : 216485,
"out" : 216485,
"queue_push_duration_in_millis" : 342466
},
"flow" : {
"input_throughput" : {
"current": 189.720,
"lifetime": 201.841
},
"filter_throughput" : {
"current": 187.810,
"lifetime": 201.799
},
"output_throughput" : {
"current": 191.087,
"lifetime": 201.761
},
"queue_backpressure" : {
"current": 0.277,
"lifetime": 0.031
},
"worker_concurrency" : {
"current": 1.973,
"lifetime": 1.721
},
"queue_persisted_growth_bytes": {
"lifetime": 0.032,
"current": 0.0
},
"queue_persisted_growth_events": {
"lifetime": -1.201,
"current": -0.9913
}
},
"plugins" : {
"inputs" : [ {
"id" : "35131f351e2dc5ed13ee04265a8a5a1f95292165-1",
"events" : {
"out" : 216485,
"queue_push_duration_in_millis" : 342466
},
"name" : "beats"
} ],
"filters" : [ {
"id" : "35131f351e2dc5ed13ee04265a8a5a1f95292165-2",
"events" : {
"duration_in_millis" : 55969,
"in" : 216485,
"out" : 216485
},
"failures" : 216485,
"patterns_per_field" : {
"message" : 1
},
"name" : "grok"
}, {
"id" : "35131f351e2dc5ed13ee04265a8a5a1f95292165-3",
"events" : {
"duration_in_millis" : 3326,
"in" : 216485,
"out" : 216485
},
"name" : "geoip"
} ],
"outputs" : [ {
"id" : "35131f351e2dc5ed13ee04265a8a5a1f95292165-4",
"events" : {
"duration_in_millis" : 278557,
"in" : 216485,
"out" : 216485
},
"name" : "elasticsearch"
} ]
},
"reloads" : {
"last_error" : null,
"successes" : 0,
"last_success_timestamp" : null,
"last_failure_timestamp" : null,
"failures" : 0
},
"queue" : {
"type" : "memory"
}
},
"test2" : {
"events" : {
"duration_in_millis" : 2222229,
"in" : 87247,
"filtered" : 87247,
"out" : 87247,
"queue_push_duration_in_millis" : 1532
},
"flow" : {
"input_throughput" : {
"current": 189.720,
"lifetime": 201.841
},
"filter_throughput" : {
"current": 187.810,
"lifetime": 201.799
},
"output_throughput" : {
"current": 191.087,
"lifetime": 201.761
},
"queue_backpressure" : {
"current": 0.871,
"lifetime": 0.031
},
"worker_concurrency" : {
"current": 4.71,
"lifetime": 1.201
}
},
"plugins" : {
"inputs" : [ {
"id" : "d7ea8941c0fc48ac58f89c84a9da482107472b82-1",
"events" : {
"out" : 87247,
"queue_push_duration_in_millis" : 1532
},
"name" : "twitter"
} ],
"filters" : [ ],
"outputs" : [ {
"id" : "d7ea8941c0fc48ac58f89c84a9da482107472b82-2",
"events" : {
"duration_in_millis" : 139545,
"in" : 87247,
"out" : 87247
},
"name" : "elasticsearch"
} ]
},
"reloads" : {
"last_error" : null,
"successes" : 0,
"last_success_timestamp" : null,
"last_failure_timestamp" : null,
"failures" : 0
},
"queue" : {
"type" : "memory"
}
}
}You can see the stats for a specific pipeline by including the pipeline ID. In
the following example, the ID of the pipeline is test:
curl -XGET 'localhost:9600/_node/stats/pipelines/test?pretty'Example response:
{
"test" : {
"events" : {
"duration_in_millis" : 365495,
"in" : 216485,
"filtered" : 216485,
"out" : 216485,
"queue_push_duration_in_millis" : 342466
},
"flow" : {
"input_throughput" : {
"current": 189.720,
"lifetime": 201.841
},
"filter_throughput" : {
"current": 187.810,
"lifetime": 201.799
},
"output_throughput" : {
"current": 191.087,
"lifetime": 201.761
},
"queue_backpressure" : {
"current": 0.277,
"lifetime": 0.031
},
"worker_concurrency" : {
"current": 1.973,
"lifetime": 1.721
},
"queue_persisted_growth_bytes": {
"lifetime": 0.032,
"current": 0.0
},
"queue_persisted_growth_events": {
"lifetime": -1.201,
"current": -0.9913
}
},
"plugins" : {
"inputs" : [ {
"id" : "35131f351e2dc5ed13ee04265a8a5a1f95292165-1",
"events" : {
"out" : 216485,
"queue_push_duration_in_millis" : 342466
},
"name" : "beats"
} ],
"filters" : [ {
"id" : "35131f351e2dc5ed13ee04265a8a5a1f95292165-2",
"events" : {
"duration_in_millis" : 55969,
"in" : 216485,
"out" : 216485
},
"failures" : 216485,
"patterns_per_field" : {
"message" : 1
},
"name" : "grok"
}, {
"id" : "35131f351e2dc5ed13ee04265a8a5a1f95292165-3",
"events" : {
"duration_in_millis" : 3326,
"in" : 216485,
"out" : 216485
},
"name" : "geoip"
} ],
"outputs" : [ {
"id" : "35131f351e2dc5ed13ee04265a8a5a1f95292165-4",
"events" : {
"duration_in_millis" : 278557,
"in" : 216485,
"out" : 216485
},
"name" : "elasticsearch"
} ]
},
"reloads" : {
"last_error" : null,
"successes" : 0,
"last_success_timestamp" : null,
"last_failure_timestamp" : null,
"failures" : 0
},
"queue": {
"type" : "persisted",
"capacity": {
"max_unread_events": 0,
"page_capacity_in_bytes": 67108864,
"max_queue_size_in_bytes": 1073741824,
"queue_size_in_bytes": 3885
},
"data": {
"path": "/pipeline/queue/path",
"free_space_in_bytes": 936886480896,
"storage_type": "apfs"
},
"events": 0,
"events_count": 0,
"queue_size_in_bytes": 3885,
"max_queue_size_in_bytes": 1073741824
}
}
}
}The following request returns a JSON document that shows info about config reload successes and failures.
curl -XGET 'localhost:9600/_node/stats/reloads?pretty'Example response:
{
"reloads": {
"successes": 0,
"failures": 0
}
}When Logstash is running in a container, the following request returns a JSON document that contains cgroup information to give you a more accurate view of CPU load, including whether the container is being throttled.
curl -XGET 'localhost:9600/_node/stats/os?pretty'Example response:
{
"os" : {
"cgroup" : {
"cpuacct" : {
"control_group" : "/elastic1",
"usage_nanos" : 378477588075
},
"cpu" : {
"control_group" : "/elastic1",
"cfs_period_micros" : 1000000,
"cfs_quota_micros" : 800000,
"stat" : {
"number_of_elapsed_periods" : 4157,
"number_of_times_throttled" : 460,
"time_throttled_nanos" : 581617440755
}
}
}
}You can monitor stats for the geoip databases used with the Geoip filter plugin.
curl -XGET 'localhost:9600/_node/stats/geoip_download_manager?pretty'For more info, see Database Metrics in the Geoip filter plugin docs.
The hot threads API gets the current hot threads for Logstash. A hot thread is a Java thread that has high CPU usage and executes for a longer than normal period of time.
curl -XGET 'localhost:9600/_node/hot_threads?pretty'The output is a JSON document that contains a breakdown of the top hot threads for Logstash.
Example response:
{
"hot_threads" : {
"time" : "2017-06-06T18:25:28-07:00",
"busiest_threads" : 3,
"threads" : [ {
"name" : "Ruby-0-Thread-7",
"percent_of_cpu_time" : 0.0,
"state" : "timed_waiting",
"path" : "/path/to/logstash-{logstash_version}/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/thread_pool.rb:187",
"traces" : [ "java.lang.Object.wait(Native Method)", "org.jruby.RubyThread.sleep(RubyThread.java:1002)", "org.jruby.RubyKernel.sleep(RubyKernel.java:803)" ]
}, {
"name" : "[test2]>worker3",
"percent_of_cpu_time" : 0.85,
"state" : "waiting",
"traces" : [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)", "java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)" ]
}, {
"name" : "[test2]>worker2",
"percent_of_cpu_time" : 0.85,
"state" : "runnable",
"traces" : [ "org.jruby.RubyClass.allocate(RubyClass.java:225)", "org.jruby.RubyClass.newInstance(RubyClass.java:856)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)" ]
} ]
}
}The parameters allowed are:
threads
|
The number of hot threads to return. The default is 10. |
stacktrace_size
|
The depth of the stack trace to report for each thread. The default is 50. |
human
|
If true, returns plain text instead of JSON format. The default is false. |
ignore_idle_threads
|
If true, does not return idle threads. The default is true. |
See Common Options for a list of options that can be applied to all Logstash monitoring APIs.
You can use the ?human parameter to return the document in a human-readable format.
curl -XGET 'localhost:9600/_node/hot_threads?human=true'Example of a human-readable response:
::: {}
Hot threads at 2017-06-06T18:31:17-07:00, busiestThreads=3:
================================================================================
0.0 % of cpu usage, state: timed_waiting, thread name: 'Ruby-0-Thread-7'
/path/to/logstash-{logstash_version}/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/thread_pool.rb:187
java.lang.Object.wait(Native Method)
org.jruby.RubyThread.sleep(RubyThread.java:1002)
org.jruby.RubyKernel.sleep(RubyKernel.java:803)
--------------------------------------------------------------------------------
0.0 % of cpu usage, state: waiting, thread name: 'defaultEventExecutorGroup-5-4'
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
--------------------------------------------------------------------------------
0.05 % of cpu usage, state: timed_waiting, thread name: '[test]-pipeline-manager'
java.lang.Object.wait(Native Method)
java.lang.Thread.join(Thread.java:1253)
org.jruby.internal.runtime.NativeThread.join(NativeThread.java:75)