Each JSON field can be mapped to a specific core type. JSON itself
already provides us with some typing, with its support for string
,
integer
/long
, float
/double
, boolean
, and null
.
The following sample tweet JSON document will be used to explain the core types:
{
"tweet" {
"user" : "kimchy"
"message" : "This is a tweet!",
"postDate" : "2009-11-15T14:12:12",
"priority" : 4,
"rank" : 12.3
}
}
Explicit mapping for the above JSON tweet can be:
{
"tweet" : {
"properties" : {
"user" : {"type" : "string", "index" : "not_analyzed"},
"message" : {"type" : "string", "null_value" : "na"},
"postDate" : {"type" : "date"},
"priority" : {"type" : "integer"},
"rank" : {"type" : "float"}
}
}
}
The text based string type is the most basic type, and contains one or more characters. An example mapping can be:
{
"tweet" : {
"properties" : {
"message" : {
"type" : "string",
"store" : true,
"index" : "analyzed",
"null_value" : "na"
},
"user" : {
"type" : "string",
"index" : "not_analyzed",
"norms" : {
"enabled" : false
}
}
}
}
}
The above mapping defines a string
message
property/field within the
tweet
type. The field is stored in the index (so it can later be
retrieved using selective loading when searching), and it gets analyzed
(broken down into searchable terms). If the message has a null
value,
then the value that will be stored is na
. There is also a string
user
which is indexed as-is (not broken down into tokens) and has norms
disabled (so that matching this field is a binary decision, no match is
better than another one).
The following table lists all the attributes that can be used with the
string
type:
Attribute | Description |
---|---|
|
The name of the field that will be stored in the index. Defaults to the property/field name. |
|
Set to |
|
Set to |
|
Set to |
|
Possible values are |
|
The boost value. Defaults to |
|
When there is a (JSON) null value for the field, use the
|
|
Boolean value if norms should be enabled or
not. Defaults to |
|
Describes how norms should be loaded, possible values are
|
|
Allows to set the indexing
options, possible values are |
|
The analyzer used to analyze the text contents when
|
|
The analyzer used to analyze the text contents when
|
|
The analyzer used to analyze the field when part of a query string. Can be updated on an existing field. |
|
Should the field be included in the |
|
The analyzer will ignore strings larger than this size.
Useful for generic |
|
Position increment gap between field instances with the same field name. Defaults to 0. |
The string
type also support custom indexing parameters associated
with the indexed value. For example:
{
"message" : {
"_value": "boosted value",
"_boost": 2.0
}
}
The mapping is required to disambiguate the meaning of the document.
Otherwise, the structure would interpret "message" as a value of type
"object". The key _value
(or value
) in the inner document specifies
the real string content that should eventually be indexed. The _boost
(or boost
) key specifies the per field document boost (here 2.0).
Norms store various normalization factors that are later used (at query time) in order to compute the score of a document relatively to a query.
Although useful for scoring, norms also require quite a lot of memory (typically in the order of one byte per document per field in your index, even for documents that don’t have this specific field). As a consequence, if you don’t need scoring on a specific field, it is highly recommended to disable norms on it. In particular, this is the case for fields that are used solely for filtering or aggregations.
coming[1.2.0] In case you would like to disable norms after the fact, it is possible to do so by using the PUT mapping API. Please however note that norms won’t be removed instantly, but as your index will receive new insertions or updates, and segments get merged. Any score computation on a field that got norms removed might return inconsistent results since some documents won’t have norms anymore while other documents might still have norms.
A number based type supporting float
, double
, byte
, short
,
integer
, and long
. It uses specific constructs within Lucene in
order to support numeric values. The number types have the same ranges
as corresponding
Java
types. An example mapping can be:
{
"tweet" : {
"properties" : {
"rank" : {
"type" : "float",
"null_value" : 1.0
}
}
}
}
The following table lists all the attributes that can be used with a numbered type:
Attribute | Description |
---|---|
|
The type of the number. Can be |
|
The name of the field that will be stored in the index. Defaults to the property/field name. |
|
Set to |
|
Set to |
|
Set to |
|
The precision step (number of terms generated for
each number value). Defaults to |
|
The boost value. Defaults to |
|
When there is a (JSON) null value for the field, use the
|
|
Should the field be included in the |
|
Ignored a malformed number. Defaults to |
|
Try convert strings to numbers and truncate fractions for integers. Defaults to |
The token_count
type maps to the JSON string type but indexes and stores
the number of tokens in the string rather than the string itself. For
example:
{
"tweet" : {
"properties" : {
"name" : {
"type" : "string",
"fields" : {
"word_count": {
"type" : "token_count",
"store" : "yes",
"analyzer" : "standard"
}
}
}
}
}
}
All the configuration that can be specified for a number can be specified
for a token_count. The only extra configuration is the required
analyzer
field which specifies which analyzer to use to break the string
into tokens. For best performance, use an analyzer with no token filters.
Note
|
Technically the |
The date type is a special type which maps to JSON string type. It
follows a specific format that can be explicitly set. All dates are
UTC
. Internally, a date maps to a number type long
, with the added
parsing stage from string to long and from long to string. An example
mapping:
{
"tweet" : {
"properties" : {
"postDate" : {
"type" : "date",
"format" : "YYYY-MM-dd"
}
}
}
}
The date type will also accept a long number representing UTC milliseconds since the epoch, regardless of the format it can handle.
The following table lists all the attributes that can be used with a date type:
Attribute | Description |
---|---|
|
The name of the field that will be stored in the index. Defaults to the property/field name. |
|
The date
format. Defaults to |
|
Set to |
|
Set to |
|
Set to |
|
The precision step (number of terms generated for
each number value). Defaults to |
|
The boost value. Defaults to |
|
When there is a (JSON) null value for the field, use the
|
|
Should the field be included in the |
|
Ignored a malformed number. Defaults to |
The boolean type Maps to the JSON boolean type. It ends up storing
within the index either T
or F
, with automatic translation to true
and false
respectively.
{
"tweet" : {
"properties" : {
"hes_my_special_tweet" : {
"type" : "boolean",
}
}
}
}
The boolean type also supports passing the value as a number or a string
(in this case 0
, an empty string, F
, false
, off
and no
are
false
, all other values are true
).
The following table lists all the attributes that can be used with the boolean type:
Attribute | Description |
---|---|
|
The name of the field that will be stored in the index. Defaults to the property/field name. |
|
Set to |
|
Set to |
|
The boost value. Defaults to |
|
When there is a (JSON) null value for the field, use the
|
The binary type is a base64 representation of binary data that can be stored in the index. The field is not stored by default and not indexed at all.
{
"tweet" : {
"properties" : {
"image" : {
"type" : "binary",
}
}
}
}
The following table lists all the attributes that can be used with the binary type:
Attribute | Description |
---|---|
|
The name of the field that will be stored in the index. Defaults to the property/field name. |
|
Set to |
It is possible to control which field values are loaded into memory, which is particularly useful for faceting on string fields, using fielddata filters, which are explained in detail in the Fielddata section.
Fielddata filters can exclude terms which do not match a regex, or which
don’t fall between a min
and max
frequency range:
{
tweet: {
type: "string",
analyzer: "whitespace"
fielddata: {
filter: {
regex: {
"pattern": "^#.*"
},
frequency: {
min: 0.001,
max: 0.1,
min_segment_size: 500
}
}
}
}
}
These filters can be updated on an existing field mapping and will take effect the next time the fielddata for a segment is loaded. Use the Clear Cache API to reload the fielddata using the new filters.
Posting formats define how fields are written into the index and how
fields are represented into memory. Posting formats can be defined per
field via the postings_format
option. Postings format are configurable.
Elasticsearch has several builtin formats:
direct
-
A postings format that uses disk-based storage but loads its terms and postings directly into memory. Note this postings format is very memory intensive and has certain limitation that don’t allow segments to grow beyond 2.1GB see \{@link DirectPostingsFormat} for details.
memory
-
A postings format that stores its entire terms, postings, positions and payloads in a finite state transducer. This format should only be used for primary keys or with fields where each term is contained in a very low number of documents.
pulsing
-
A postings format that in-lines the posting lists for very low frequent terms in the term dictionary. This is useful to improve lookup performance for low-frequent terms.
bloom_default
-
A postings format that uses a bloom filter to improve term lookup performance. This is useful for primary keys or fields that are used as a delete key.
bloom_pulsing
-
A postings format that combines the advantages of bloom and pulsing to further improve lookup performance.
default
-
The default Elasticsearch postings format offering best general purpose performance. This format is used if no postings format is specified in the field mapping.
On all field types it possible to configure a postings_format
attribute:
{
"person" : {
"properties" : {
"second_person_id" : {"type" : "string", "postings_format" : "pulsing"}
}
}
}
On top of using the built-in posting formats it is possible define custom postings format. See codec module for more information.
Doc values formats define how fields are written into column-stride storage in the index for the purpose of sorting or faceting. Fields that have doc values enabled will have special field data instances, which will not be uninverted from the inverted index, but directly read from disk. This makes _refresh faster and ultimately allows for having field data stored on disk depending on the configured doc values format.
Doc values formats are configurable. Elasticsearch has several builtin formats:
memory
-
A doc values format which stores data in memory. Compared to the default field data implementations, using doc values with this format will have similar performance but will be faster to load, making '_refresh' less time-consuming.
disk
-
A doc values format which stores all data on disk, requiring almost no memory from the JVM at the cost of a slight performance degradation.
default
-
The default Elasticsearch doc values format, offering good performance with low memory usage. This format is used if no format is specified in the field mapping.
On all field types, it is possible to configure a doc_values_format
attribute:
{
"product" : {
"properties" : {
"price" : {"type" : "integer", "doc_values_format" : "memory"}
}
}
}
On top of using the built-in doc values formats it is possible to define custom doc values formats. See codec module for more information.
Elasticsearch allows you to configure a similarity (scoring algorithm) per field.
The similarity
setting provides a simple way of choosing a similarity algorithm
other than the default TF/IDF, such as BM25
.
You can configure similarities via the similarity module
Defining the Similarity for a field is done via the similarity
mapping
property, as this example shows:
{
"book" : {
"properties" : {
"title" : { "type" : "string", "similarity" : "BM25" }
}
}
The following Similarities are configured out-of-box:
default
-
The Default TF/IDF algorithm used by Elasticsearch and Lucene in previous versions.
BM25
-
The BM25 algorithm. See Okapi_BM25 for more details.
added[1.0.0.RC2]
Adding copy_to
parameter to any field mapping will cause all values of this field to be copied to fields specified in
the parameter. In the following example all values from fields title
and abstract
will be copied to the field
meta_data
.
{
"book" : {
"properties" : {
"title" : { "type" : "string", "copy_to" : "meta_data" },
"abstract" : { "type" : "string", "copy_to" : "meta_data" },
"meta_data" : { "type" : "string" },
}
}
Multiple fields are also supported:
{
"book" : {
"properties" : {
"title" : { "type" : "string", "copy_to" : ["meta_data", "article_info"] },
}
}
added[1.0.0.RC1]
The fields
options allows to map several core types fields into a single
json source field. This can be useful if a single field need to be
used in different ways. For example a single field is to be used for both
free text search and sorting.
{
"tweet" : {
"properties" : {
"name" : {
"type" : "string",
"index" : "analyzed",
"fields" : {
"raw" : {"type" : "string", "index" : "not_analyzed"}
}
}
}
}
}
In the above example the field name
gets processed twice. The first time it gets
processed as an analyzed string and this version is accessible under the field name
name
, this is the main field and is in fact just like any other field. The second time
it gets processed as a not analyzed string and is accessible under the name name.raw
.
The include_in_all
setting is ignored on any field that is defined in
the fields
options. Setting the include_in_all
only makes sense on
the main field, since the raw field value to copied to the _all
field,
the tokens aren’t copied.
In the essence a field can’t be updated. However multi fields can be
added to existing fields. This allows for example to have a different
index_analyzer
configuration in addition to the already configured
index_analyzer
configuration specified in the main and other multi fields.
Also the new multi field will only be applied on document that have been added after the multi field has been added and in fact the new multi field doesn’t exist in existing documents.
Another important note is that new multi fields will be merged into the list of existing multi fields, so when adding new multi fields for a field previous added multi fields don’t need to be specified.
deprecated[1.0.0,Use copy_to
instead]
The multi fields defined in the fields
are prefixed with the
name of the main field and can be accessed by their full path using the
navigation notation: name.raw
, or using the typed navigation notation
tweet.name.raw
. The path
option allows to control how fields are accessed.
If the path
option is set to full
, then the full path of the main field
is prefixed, but if the path
option is set to just_name
the actual
multi field name without any prefix is used. The default value for
the path
option is full
.
The just_name
setting, among other things, allows indexing content of multiple
fields under the same name. In the example below the content of both fields
first_name
and last_name
can be accessed by using any_name
or tweet.any_name
.
{
"tweet" : {
"properties": {
"first_name": {
"type": "string",
"index": "analyzed",
"path": "just_name",
"fields": {
"any_name": {"type": "string","index": "analyzed"}
}
},
"last_name": {
"type": "string",
"index": "analyzed",
"path": "just_name",
"fields": {
"any_name": {"type": "string","index": "analyzed"}
}
}
}
}
}