Repository navigation
Getting Started
- Install and Run
- Basic Operations 2. Indexing a document 2. Retrieving a document 2. Deleting a document 2. Checking the existence of a document 2. Getting all documents in an index of a certain type 2. Creating simple queries (Search Lite) 2. Search using Query DSL 2. Filtering 2. Full text search
- About shards and scoring
The following paragraphs will describe how to install and run Elasticsearch locally. They will also provide examples on the basics of working with it. As this guide only provides brief information on getting up to speed with Elasticsearch please look at the official guide that can be found here for a more detailed approach on Elasticsearch. For even more in-depth information check the Elsticsearch reference.
Prerequisites
- A recent version of Java, preferably the latest official version
You can get your hands on a copy of Elasticsearch here. Download the archive for your platform, extract it and you are now ready to run Elasticsearch. Elastisearch comes with sensible configuration defaults so there is no need to change anything at this point.
In order to run Elasticsearch navigate to the folder where the archive was extracted and depending on the platform run
-
./bin/elasticsearchfor UNIX flavors ( you can add the -d flag to run it as daemon) -
\bin\elasticsearch.batfor Windows
To test the installation open another terminal and run curl 'http://localhost:9200/?pretty'. For Windows installations you can download cURL from here. You should receive a response like the following:
{
"name" : "Boom Boom",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "IxDW7YZhRTqm0sRlJfEAcw",
"version" : {
"number" : "2.4.1",
"build_hash" : "c67dc32e24162035d18d6fe1e952c4cbcbe79d16",
"build_timestamp" : "2016-09-27T18:57:55Z",
"build_snapshot" : false,
"lucene_version" : "5.5.2"
},
"tagline" : "You Know, for Search"
}
This means Elasticsearch is up and running and you can start playing with it.
Elasticsearch exposes a REST API using the HTTP protocol and JSON format that can be used to communicate with it and perform various operations. It also provides clients for the popular languages that can be used for the same purposes. These clients won't be covered in this guide. Please note that JSON is the serialization format used by Elasticsearch.
To index a document means to store it in and it is done by doing an HTTP PUT request with the document.
curl -X PUT 'http://localhost:9200/university/student/1' -d
'{
"first_name":"John",
"last_name":"Smith"
}'
This stores the JSON in an index called university (first element in path). The type of the document is student (second element in path) and its id will be 1 (third element in path). If a document already exists it will be overwritten by the new one.
A response is returned and a successful one looks like the following
{
"_index":"university",
"_type":"student",
"_id":"1",
"_version":1,
"_shards":{
"total":2,
"successful":1,
"failed":0
},
"created":true
}
There is a solution that prevents the overwrite of the document if it exists. One way is to add to the path the _create element and the other is to add the query parameter op_type=create. The URL for both cases would like this:
http://localhost:9200/university/student/1/_create, respectively http://localhost:9200/university/student/1?op_type=create
In order to obtain a document an HTTP GET request needs to performed. The URL is constructed in a similar fashion as the one used to index the document. The template for the URL looks like this: {ELASTICSEARCH_BASE_URL}/{INDEX}/{TYPE}/{ID}. For example:
curl -X GET 'http://localhost:9200/university/student/1'
The response for the query looks something like this
{
"_index":"university",
"_type":"student",
"_id":"1",
"_version":1,
"found":true,
"_source":{
"first_name":"John",
"last_name":"Smith"
}
}
In order to delete a document an HTTP DELETE request needs to performed. The URL is constructed in a similar fashion as the one used to index the document. The template for the URL looks like this: {ELASTICSEARCH_BASE_URL}/{INDEX}/{TYPE}/{ID}. For example:
curl -X DELETE 'http://localhost:9200/university/student/1'
The response for the query looks something like this
{
"found":true,
"_index":"university",
"_type":"student",
"_id":"1",
"_version":2,
"_shards":{
"total":2,
"successful":1,
"failed":0
}
}
In order to obtain a document an HTTP HEAD request needs to performed. The URL is constructed in a similar fashion as the one used to index the document. The template for the URL looks like this: {ELASTICSEARCH_BASE_URL}/{INDEX}/{TYPE}/{ID}. Since a HEAD request only returns headers use the -i flag for curl to see the response status and headers. For example:
curl -i -X HEAD 'http://localhost:9200/university/student/7'
Getting all documents in an index of a certain type is similar to getting a single document but instead of specifying an id, _search is used instead at the end of the path. For example:
curl -X GET 'http://localhost:9200/university/student/_search'
This request will list all the documents in the university index with that are of the student type.
Performing a simple query is done in a similar way to getting all the documents in an index of a certain type. The difference is that search criteria are added as a query parameter with the key q. For example:
url -X GET 'http://localhost:9200/university/student/_search?q=last_name:Smith'
This query returns the documents in the university index that are of student type that have field last_name with the value Smith.
For more complex queries Elastisearch provides a query language called Query DSL. In order to make use of it a request is performed in a similar way to getting all the documents in an index of a certain type. The difference is that the query is specified in a JSON request body.
curl -XGET 'localhost:9200/university/student/_search?pretty' -d'
{
"query" : {
"match" : {
"last_name" : "Smith"
}
}
}'
This query returns the documents in the university index that are of student type that have field last_name with the value Smith.
With the Query DSL it is possible to narrow down the results returned. This is done in a similar way to searching with Query DSL by adding to it a filter element which specified the criteria used. For example:
curl -X GET 'http://localhost:9200/university/student/_search?pretty' -d'
{
"query" : {
"bool" : {
"must" : {
"match" : {
"last_name" : "Smith"
}
},
"filter" : {
"range" : {
"average" : { "gt" : 5 }
}
}
}
}
}'
Assuming that the students have a field average the above query retrieves all the students that have the last name Smith that have an average greater than 5.
This is where Elasticsearch differentiates between the traditional databases. It allows searching for different terms and returns relevance scores. For example:
curl -X GET 'http://localhost:9200/university/students/_search?pretty' -d'
{
"query" : {
"match" : {
"about" : "cook skier"
}
}
}'
In our example Elasticsearch returns the following 2 documents:
{
...
"hits": {
"total": 2,
"max_score": 0.16273327,
"hits": [
{
...
"_score": 0.16273327,
"_source": {
"first_name": "John",
"last_name": "Smith",
"average": 8,
"about": "cook, skier, wine drinker",
}
},
{
...
"_score": 0.016878016,
"_source": {
"first_name": "Jane",
"last_name": "Smith",
"average": 9,
"about": "I like to cook traditional recipes",
}
}
]
}
}
As you can see two documents have been returned but one of them includes only one of the searched terms. This is strange, you may think but the second document is returned because it contains one of the terms searched and Elasticsearch differentiates between them by using a score that indicates the relevance of the document in the context of the search. This is the reason Jane Smith has a lower score than John Smith.
Elasticsearch distributes the documents inside an index in shards. Replicas are copies of shards, and they are primarily for improving search performance. They can be added or removed at any time, while the number of total shards is specified at index creation. Only primary shards (not replicas) can accept indexing requests, but any shard can accept search requests. Distribution of replica shards is different from that of primary shards over the nodes.
Elasticsearch uses term frequency statistics (document frequency, inverse document frequency and field-length norm) to calculate relevance. These statistics are computed for each individual shard. A search request is received by a node, which forwards it to all of the possible shards (to a primary or its replica), the best results are returned by each of them, then the original node aggregates the results and returns the final set. If you want to understand this mechanism better, please read the corresponding chapter from the official elasticsearch documentation.
If you want more details on how the scores are calculated, please check this topic too.
Allocating shards in an incorrect manner can lead to poor document relevance. This problem is discussed in Troubleshooting.