Repository navigation
Amazon Elasticsearch Service
- Recommended configuration 2. Disable Swapping 2. Heap Memory 2. File Descriptors 2. Assigning Names 2. Paths 2. I/O Scheduler 2. Enable Slowlog
- Logging, Alerting and Disaster Recovery 2. Logging 2. Alerting 2. Auto Scaling 2. Disaster Recovery
- Limitations
- Backup-Restore Mechanism 2. Using the amazon service 2. [Manual Backup-Restore] (#manual-backup-restore) 3. [Prerequisites] (#prerequisites) 3. [Backup] (#backup) 3. [Restore] (#restore)
This page is meant to offer an overview on how Amazon Elasticsearch Service can be used, how can it be configured the way Elastic recommends to and other useful information about it.
Elastic provides a guide on how to configure the environment for ES in order to run smoothly.
Elastic recommends to disable swapping memory to disk. As we do not have a means of checking this we rely on the Amazon documentation.
Elastic recommends to have 50% of the available RAM allocated to the Elasticsearch Heap but no more than 32GB. AES does this automatically. This can be checked by performing a GET _nodes/stats and for each node under the JVM element the heap_max_in_bytes field indicates how much is allocated.
Elastic recommends to increase the number of file descriptors to a large value (in their example 64 000). AES sets this number to 128 000. This can be checked by performing a GET _nodes/stats and for each node, under the process element the max_file_descriptors field indicates the number of file descriptors.
Elastic recommends to change the name of nodes and the name of the cluster. AES does this out of the box. The name of the cluster is formed by prepending a number and a colon to the domain name (e.g. 371548805176:concept-search-mvp). The only part that is under our control is the name of the domain. Each node is given a different random name that is easy to read for a human (e.g. Night Thrasher).
Elastic recommends to change the paths where ES places data, logs and plugins to a different path than installation directory which is the default.
!!! Unable to figure out if AES does this or not. These properties are not returned by any of the endpoints and we are unable to check on the machine. ASSUMPTION: Amazon takes proper care of this!
Elastic recommends to change the I/O scheduler to NOOP or deadline.
!!! This property is not returned by any of the endpoints and there is no way to access the machines to check this. ASSUMPTION: Amazon takes proper care of this!
When suspecting that queries and indexing become a bottleneck ES allow the enable of slowlog. It can be done easily by setting the desired log level and thresholds at the my_index/settings endpoint. However there is no known way to access these logs.
Amazon Elasticsearch Service does not expose ES internal logs which among others means no slowlogs (even though they can be enabled. It only exposes the logs generated by the configuration API exposed by Amazon for creating, configuring, deleting the domain and for tags. For more information please go to the Amazon's official docs.
Amazon Cloudwatch enables users to create alerts based on the metrics it offers which range from the status of the cluster (red, yellow, green) to CPU usage, storage space, JVM memory, write/read throughput. Creating an alarm is as easily as selecting the metric and choosing a threshold. When the condition set is met an alarm is triggered. At the trigger of the alarm there are two categories of actions that can take place: notification and auto scaling. Notifications sent cannot be customized to follow a certain template. However, the message contains a lot of information about the triggered alarm.
Another solution might be to import the data from the Cloudwatch metrics into Splunk and do the alerting part similar to the existing ones. This approach would enable custom notifications. The Integration Engineers cover the part to get the metrics from Cloudwatch to Splunk (~ 1/2 - 1 day) and all what remains to do is to create the custom alarms.
Scaling should not be done in automated way as this quora question and this stackoverflow question indicate. This is due to the fact that sharding and reallocation are very intensive tasks and may degrade performance even more if not done in a supervised manner.
AES automatically takes care of managing the cluster and detects and replaces ES nodes that have failed, according to their official documentation.
AES features automatic snapshots to back-up and restore ES domains.
Amazon provides a page in which it describes the limitations of its Elasticsearch Service.
It provides another page with the supported operations.
There are two ways to backup or restore the Amazon Elasticsearch service
By default, Amazon Elasticsearch Service will automatically create daily snapshots of each Amazon Elasticsearch domain. Amazon Elasticsearch Service will retain the last 14 days worth of automated daily snapshots. In order to restore to a snapshot, Amazon must be contacted and asked to do the restore.
An Amazon S3 Bucket must be registered to the Elastic Search Service as a repository for backups named index-backups. For more information check the Amazon [official documentation] (http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains.html).
Run the following command to manually take a snapshot:
curl -XPUT 'http://<Elasticsearch_domain_endpoint>/_snapshot/index-backups/snapshot_name'
This commands creates a snapshot with the given name in the index-backups repository .
Note The time required to take a snapshot increases with the size of the Amazon ES domain. Long-running snapshot operations commonly encounter the following error: BotoServerError: 504 GATEWAY_TIMEOUT. Typically, you can ignore these errors and wait for the operation to complete successfully. Use the following command to verify the state of all snapshots of your domain:
curl -XGET 'http://<Elasticsearch_domain_endpoint>/_snapshot/index-backups/_all?pretty'
In order to restore to a snapshot, all open indices in the Amazon ES domain must be deleted or renamed.
You cannot restore a snapshot of your indices to an Elasticsearch cluster that already contains indices with the same names. Currently, Amazon ES does not support the Elasticsearch _close API, so you must use one of the following alternatives:
- Delete the indices on the same Amazon ES domain, then restore the snapshot.
- Restore the snapshot to a different Amazon ES domain
The following command deletes all existing indices:
curl -XDELETE 'http://<Elasticsearch_domain_endpoint>/_all'
When there are no open indices with the same name in the ES, run the following command to restore a snapshot:
curl -XPOST 'http://<Elasticsearch_domain_endpoint>/_snapshot/index-backups/snapshot_name/_restore'
This command restores to the snapshot with the given name from the index-backups repository.