Repository navigation
Monitoring and Deployment
Elastic offers a tool for monitoring ES called Marvel which is free to use even in production. ES also offers an API for monitoring which will be discussed further.
For a high level overview on the status of the ES cluster an HTTP GET request needs to be performed at the _cluster/health endpoint. An example with curl:
curl -X GET 'http://localhost:9200/_cluster/health
In order to make the examples more cleaner only the HTTP method along with the relative path to the ES cluster of the endpoint will be used from now on. The above example depicted in this way:
GET _cluster/health
The JSON response contains a status field that may have the following values:
- green - All primary and replica shards are allocated. Your cluster is 100% operational
- yellow - All primary shards are allocated, but at least one replica is missing. No missing data but if we lose shards we will lose data
- red - At least one primary shard (and all of its replicas) is missing. This means you are missing data
For monitoring the indices of the cluster simply append the query parameter level=indices and a detailed status of the indices is returned.
GET _cluster/health?level=indices
An even more detailed status can be obtained at a shard index by appending the query parameter level=shards. The output is verbose and might be difficult to work with due to its size as it contains information about every shard on every index.
GET _cluster/health?level=shards
Another endpoint that may prove useful, especially when running integration tests, is the one that blocks until the status of the cluster turns green.
GET _cluster/health?wait_for_status=green
There are times that require to get the status of an individual node. ES provides an endpoint for this as well
GET _nodes/stats
It also provides information about the JVM and there are a few things that are worth mentioning. When the heap usage percent starts to go over 75% performance may decrease significantly and steps to mitigate this should be taken. A few suggestions are to enable slow GC logging and to limit memory usage with the circuit breaker.
In order to see all the information provided by the stats endpoint but at a the cluster level ES exposes an endpoint for this. It aggregates all the metrics provided by the /_nodes/stats endpoint under the same umbrella.
GET _cluster/stats
ES also offers endpoints for stats from the indices point of view.
GET my_index/_stats
Returns the stats of the index my_index.
GET my_index,another_index/_stats
Returns the stats of 2 indices: my_index and another_index. Stats for multiple indices can be returned by separating their names with comma.
GET _all/_stats
Returns stats for all indices in the cluster.
Although rare, there are cases when ES is not able to process all tasks and the master node becomes a bottle neck. To see if this is the case you can check the pending tasks endpoint.
GET _cluster/pending_tasks
ES offers the cat API, named after the linux command. It presents the data in a tabular format instead of JSON. To check the available endpoints perform a GET /_cat and it proves to be very useful for system administrators as the data is provided in a familiar way and easy to read way.
Please visit the ES official docs for a comprehensive description of the cat API. In this guide a list of one of the most used endpoints is provided:
GET /_cat/health
GET /_cat/health?v
GET /_cat/nodes?help
GET /_cat/nodes?v&h=ip,port,heapPercent,heapMax
This section is a collection of recommendations from ES official guide for hardware specification and configurations.
RAM
At the time of writing ES work well with 32 GB of RAM, even with 16 and ideally this value is 64. After 64 GB a myriad of problems may appear due to some limitations.
It is recommended to give ES a heap size equal to half of the amount of RAM but no more than 32 GB. You can limit the heap size either with JVM flags or by setting the ES_HEAP_SIZE environment variable to the desired value (e.g. export ES_HEAP_SIZE=10g). When using the former option ensure that both min and max values are the same to prevent heap resizing as this process is very expensive.
Disable swapping as it kills performance.
Storage
An SSD is preferred with deadline or noop OS I/O scheduler.
JVM
It is recommended to use the latest version of Java and both Oracle and OpenJDK should work. Do NOT tweak JVM settings, garbage collector settings nor threadpools.
Configuration
ES comes with very good defaults, performance-wise, for the configuration. As a consequence when having these kind of issues the solution, in most cases, is to add more nodes or to improve the layout of data.
There are also other kinds of configurations besides the performance ones that are recommended to be changed.
- the name of the cluster -
cluster.name: production - the name of a node -
node.name: node_001 - where data is stored - by default data is stored in the ES install directory. It is recommended to change the following properties:
path.data,path.logsandpath.plugins - minimum master nodes - it is important for stability and preventing split-brain issues -
discovery.zen.minimum_master_nodes - recovery settings -
gateway.recover_after_nodes,gateway.expected_nodes,gateway.recover_after_time
ES offers two ways of changing configurations:
- configuration files - it requires a restart for changes to take effect
- via the configuration API - no restart required
Miscellaneous
File descriptor size needs to be set to a large value (e.g. 64000).
There are two types of clients one called the transport client which decouples the application from the cluster and is much lighter than the node client which is part of the cluster but does not hold any data.
After the cluster is up and running it is recommended to change configurations using the API provided as it does not require a restart. Changing configurations via the API is done in the following manner:
PUT /_cluster/settings
{
"persistent" : {
"discovery.zen.minimum_master_nodes" : 2
},
"transient" : {
"indices.store.throttle.max_bytes_per_sec" : "50mb"
}
}
In the request above there are 2 types of changes:
- transient - the change will be lost after a full cluster restart
- persistent - the change will survive a full cluster restart and overrides the static configuration file
The default logging level in ES is INFO, but this can be easily changed in the following manner:
PUT /_cluster/settings
{
"transient" : {
"logger.discovery" : "DEBUG"
}
}
There is another log called slowlog and that is designed to catch queries and indexing that takes more time than expected. The thresholds are configured at the index level.
PUT /my_index/_settings
{
"index.search.slowlog.threshold.query.warn" : "10s",
"index.search.slowlog.threshold.fetch.debug": "500ms",
"index.indexing.slowlog.threshold.index.info": "5s"
}
Thresholds can be configured in the elasticsearch.yml file. Once the thresholds are set the log level can be configured just like any other log.
PUT /_cluster/settings
{
"transient" : {
"logger.index.search.slowlog" : "DEBUG",
"logger.index.indexing.slowlog" : "WARN"
}
}
Another thing worth mentioning is that when doing large imports disable indexing by setting index.refresh_interval to -1.
When a node leaves the cluster the shards are being reallocated and if the node recovers in a timely manner this reallocation can cause more harm than good so it may be a good idea to delay the shard reallocation.
PUT /_all/_settings
{
"settings": {
"index.unassigned.node_left.delayed_timeout": "5m"
}
}
ES offers a way to perform rolling restarts. This means keeping the cluster up and running while taking nodes off one at a time. Disable shard reallocation, take the node down, perform maintenance, put the node back up, enable shard allocation.
PUT /_cluster/settings
{
"transient" : {
"cluster.routing.allocation.enable" : "none"
}
}
This setting disables shard reallocation and by setting the value to all it is reenabled.