Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add finer control over _source retrieval, in get, mget, get_source, explain & search API #3301

Closed
bleskes opened this issue Jul 9, 2013 · 0 comments
Closed

Add finer control over _source retrieval, in get, mget, get_source, explain & search API #3301

bleskes opened this issue Jul 9, 2013 · 0 comments
Assignees
@bleskes
Labels
>feature v1.0.0.Beta1

Comments

@bleskes
Copy link
Contributor

bleskes commented Jul 9, 2013

At the moment all of the above API offer the field parameter to retrieve part of the stored documents. However, the fields option was built to expose Lucene's stored fields and thus has some limitations when use to extract data from _source. The most important one is potentially flatting the document structure.

This feature adds a new parameter that allows directly retrieving parts of the _source, without conforming to the store fields structure.

To maintain backward compatibility, you can still retrieve the _source by specifying fields=["_source"] but this special treatment will be removed in the future.

Get API

The Get api parameters are supplied via the query string. New _source,_source_include & _source_exclude parameters are added, according to the following:

A flag to control _source retrieval

curl -XGET 'http://localhost:9200/index/type/1?_source=false'

or (default)

curl -XGET 'http://localhost:9200/index/type/1?_source=true'

Only retrieve part of the source

curl -XGET 'http://localhost:9200/index/type/1?_source=title,author'

or

curl -XGET 'http://localhost:9200/index/type/1?_source_include=title,content&_source_exclude=content.full_text'

Multi Get API

The Multi Get API allows you to control _source both on the query string (same syntax as the get API) or on a per document basis.

Query String defaults

curl -XGET 'http://localhost:9200/index/type/_mget?_source=false' -d'{
  ids: [1, 2, 3]
}'

or

curl -XGET 'http://localhost:9200/index/type/_mget?_source_include=title,content&_source_exclude=content.full_text' -d'{
  ids: [1, 2, 3]
}'

etc.

Per document settings

curl -XGET 'http://localhost:9200/_mget' -d '{
    docs: [
        { "_index": "test" , _type: "type1", "_id": "1", "_source": false },
        { "_index": "test" , _type: "type1", "_id": "2", "_source": "title" },
        { "_index": "test" , _type: "type1", "_id": "3", "_source": [ "title", "author" ] },
        { "_index": "test" , _type: "type1", "_id": "4", 
          "_source": { "include": "content" , "exclude" : "content.full_text" }  
        },
        { "_index": "test" , _type: "type1", "_id": "5", 
          "_source": { "include": [ "title", "content" ] , "exclude" : [ "content.full_text" ]}  
        }
    ]
}'

Get_source API

The get/_source API is an API that is already dedicated for _source retrieval. As such, it has a slightly different parameter naming:

curl -XGET 'http://localhost:9200/index/type/1/_source?include=title,content&exclude=content.full_text'

Explain API

The explain API also offers the fields parameter. It is now extend with query string parameters, just like the get API:

curl -XPOST 'http://localhost:9200/index/type/1/_explain?_source=false' -d'{
    "query" : { "term" : { "message" : "search" } }
}'

or

curl -XPOST 'http://localhost:9200/index/type/1/_explain?_source=title,author' -d'{
    "query" : { "term" : { "message" : "search" } }
}'

and

curl -XPOST 'http://localhost:9200/index/type/1/_explain?_source_include=title,content&_source_exclude=content.full_text' -d'{
    "query" : { "term" : { "message" : "search" } }
}'

Search API

The search API was added an extra _source key in the body, with the same options as all the above:

curl -XPOST 'http://localhost:9200/_search' -d'{
    "query" : { "term" : { "message" : "search" } },
    "_source" : false
}'

and

curl -XPOST 'http://localhost:9200/_search' -d'{
    "query" : { "term" : { "message" : "search" } },
    "_source" : "title"
}'

curl -XPOST 'http://localhost:9200/_search' -d'{
    "query" : { "term" : { "message" : "search" } },
    "_source" : [ "title" , "author" ]
}'

curl -XPOST 'http://localhost:9200/_search' -d'{
    "query" : { "term" : { "message" : "search" } },
    "_source": { "include": [ "title", "content" ] , "exclude" : [ "content.full_text" ]} 
}'

Also the search API supports accepting _source retrieval settings as query string parameters. The format is identical to the get API: _source, _source_include & _source_exclude. In the case where the parameters are supplied both in the request body and the query string, the query string parameter override the body.
@ghost ghost assigned bleskes Jul 9, 2013
@bleskes bleskes mentioned this issue Jul 9, 2013
Closed
@bleskes bleskes mentioned this issue Jul 16, 2013
Closed
bleskes added a commit to bleskes/elasticsearch that referenced this issue Jul 29, 2013
@bleskes
Added source fetching and filtering parameters to search, get, multi-…
593a62e 
…get, get-source and explain requests

Closes elastic#3301
@bleskes bleskes mentioned this issue Jul 30, 2013
Closed
@karmi karmi mentioned this issue Aug 1, 2013
Closed
@bleskes bleskes mentioned this issue Oct 11, 2013
Closed
@brusic brusic mentioned this issue Oct 14, 2013
Closed
@javanna javanna mentioned this issue Oct 18, 2013
Closed
karmi added a commit to elastic/elasticsearch-ruby that referenced this issue Oct 22, 2013
@karmi
[API] Added support for source filtering across the API
4b84bde 
Related:

* elastic/elasticsearch#1413
* elastic/elasticsearch#2951
* elastic/elasticsearch#3301
bleskes added a commit that referenced this issue Nov 26, 2013
@bleskes
[Docs] Added _source filtering to documentation
c63d8c4 
Relates to #3301
@clintongormley clintongormley mentioned this issue Jul 8, 2014
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bleskes bleskes

Labels
>feature v1.0.0.Beta1
Projects
None yet
Milestone
No milestone
1 participant
@bleskes