Amazon Elasticsearch Service (Amazon ES) is a managed service that makes it easy to deploy, operate, and scale Elasticsearch clusters in the AWS Cloud. Elasticsearch is a popular open-source search and analytics engine for use cases such as log analytics, real-time application monitoring, and clickstream analysis. With Amazon ES, you get direct access to the Elasticsearch APIs; existing code and applications work seamlessly with the service.
Amazon ES provisions all the resources for your Elasticsearch cluster and launches it. It also automatically detects and replaces failed Elasticsearch nodes, reducing the overhead associated with self-managed infrastructures. You can scale your cluster with a single API call or a few clicks in the console.
To get started using Amazon ES, you create a domain. An Amazon ES domain is synonymous with an Elasticsearch cluster. Domains are clusters with the settings, instance types, instance counts, and storage resources that you specify.
- Features of Amazon Elasticsearch Service
- Supported Elasticsearch Versions
- Pricing for Amazon Elasticsearch Service
- Getting Started with Amazon Elasticsearch Service
- Related Services
Amazon ES includes the following features:
- Numerous configurations of CPU, memory, and storage capacity, known as instance types
- Up to 1.5 PB of instance storage
- Amazon EBS storage volumes
- AWS Identity and Access Management (IAM) access control
- Easy integration with Amazon VPC and VPC security groups
- Encryption of data at rest
- Amazon Cognito authentication for Kibana
- Multiple geographical locations for your resources, known as regions and Availability Zones
- Dedicated master nodes to offload cluster management tasks
- Automated snapshots to back up and restore Amazon ES domains
- Cluster node allocation across two Availability Zones in the same region, known as zone awareness
Integration with Popular Services
- Data visualization using Kibana
- Integration with Amazon CloudWatch for monitoring Amazon ES domain metrics and setting alarms
- Integration with AWS CloudTrail for auditing configuration API calls to Amazon ES domains
- Integration with Amazon S3, Amazon Kinesis, and Amazon DynamoDB for loading streaming data into Amazon ES
Amazon ES currently supports the following Elasticsearch versions:
Compared to earlier versions of Elasticsearch, the 6.x versions offer powerful features that make them faster, more secure, and easier to use. Here are some highlights:
- Index splitting – If an index outgrows its original number of shards, the
_splitAPI offers a convenient way to split each primary shard into two or more shards in a new index.
- Vega visualizations – Kibana 6.2 and newer support the Vega visualization language, which lets you make context-aware Elasticsearch queries, combine multiple data sources into a single graph, add user interactivity to graphs, and much more.
- Ranking evaluation – The
_rank_evalAPI lets you measure and track how ranked search results perform against a set of queries to ensure that your searches perform as expected.
- Composite aggregations – These aggregations build composite buckets from one or more fields and sort them in "natural order" (alphabetically for terms, numerically or by date for histograms).
- Higher indexing performance – Newer versions of Elasticsearch provide superior indexing capabilities that significantly increase the throughput of data updates.
- Better safeguards – The 6.x versions of Elasticsearch offer many safeguards that are designed to prevent overly broad or complex queries from negatively affecting the performance and stability of the cluster.
- Kibana autocomplete – Kibana 6.3 and newer support autocomplete for queries, which greatly improves the day-to-day user experience.
For more information about the differences between Elasticsearch versions and the APIs that Amazon ES supports, see Supported Elasticsearch Operations.
If you start a new Elasticsearch project, we strongly recommend that you choose the latest supported Elasticsearch version. If you have an existing domain that uses an older Elasticsearch version, you can choose to keep the domain or migrate your data. For more information, see Upgrading Elasticsearch.
With AWS, you pay only for what you use. For Amazon ES, you pay for each hour of use of an EC2 instance and for the cumulative size of any EBS storage volumes attached to your instances. Standard AWS data transfer charges also apply.
However, a notable data transfer exception exists. If you use zone awareness, Amazon ES does not bill for traffic between the two Availability Zones in which the domain resides. Significant data transfer occurs within a domain during shard allocation and rebalancing. Amazon ES neither meters nor bills for this traffic.
If you qualify for the AWS Free Tier, you receive up to 750 hours per month of use with the
t2.small.elasticsearch instance types. You also receive up to 10 GB of Amazon EBS storage (Magnetic or General Purpose). For more information, see AWS Free Tier.
Throughout this guide, 1 MB refers to 10242 or 1,048,576 bytes. Likewise, 1 GB refers to 10243 or 1,073,741,824 bytes.
To get started, sign up for an AWS account if you don't already have one. After you are set up with an account, complete the Getting Started tutorial for Amazon Elasticsearch Service. Consult the following introductory topics if you need more information while learning about the service:
- Create a domain
- Size your domain appropriately
- Control access to your domain
- Index data manually or from other services
- Use Kibana to search your data
Amazon ES commonly is used with the following services:
Use AWS CloudTrail to get a history of the Amazon ES configuration API calls and related events for your account. For more information, see Logging Amazon Elasticsearch Service Configuration API Calls with AWS CloudTrail.
An Amazon ES domain automatically sends metrics to Amazon CloudWatch so that you can gather and analyze performance statistics. For more information, see Monitoring Cluster Metrics and Statistics with Amazon CloudWatch (Console).
CloudWatch Logs can also go the other direction. You might configure CloudWatch Logs to stream data to Amazon ES for analysis. To learn more, see Loading Streaming Data into Amazon ES from Amazon CloudWatch.
Kinesis is a managed service that scales elastically for real-time processing of streaming data at a massive scale. For more information, see Loading Streaming Data into Amazon ES from Amazon Kinesis Data Streams and Loading Streaming Data into Amazon ES from Amazon Kinesis Data Firehose.
Amazon Simple Storage Service (Amazon S3) provides storage for the internet. Amazon ES provides Lambda sample code for integration with Amazon S3. For more information, see Loading Streaming Data into Amazon ES from Amazon S3.
AWS Identity and Access Management (IAM) is a web service that you can use to manage for your Amazon ES domains. For more information, see the IAM documentation and Amazon Elasticsearch Service Access Control.
AWS Lambda is a compute service that lets you run code without provisioning or managing servers. Amazon ES provides Lambda sample code to stream data from DynamoDB, Amazon S3, and Kinesis. For more information, see Loading Streaming Data into Amazon Elasticsearch Service.
Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. To learn more, see Loading Streaming Data into Amazon ES from Amazon DynamoDB.