Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow for executing queries based on pre-defined templates #4879

Closed
MaineC opened this issue Jan 24, 2014 · 6 comments
Closed

Allow for executing queries based on pre-defined templates #4879

MaineC opened this issue Jan 24, 2014 · 6 comments
Labels
>feature :Search/Search Search-related issues that do not fall into other categories v1.1.0 v2.0.0-beta1

Comments

@MaineC
Copy link

MaineC commented Jan 24, 2014

It would be nice to be able to store pre-defined query templates that can be referenced and filled with parameter values at query time. This can be in particular useful to be able to quickly replay queries with slightly different templates but identical parameter values e.g. in order to compare slightly different ways to formulate Elasticsearch queries wrt. runtime performance and actual results returned.

For developing and testing the template, both, template_string and template_vars can be submitted as part of the search request:

GET _search
{
    "query": {
        "template": {
            "template_string": "{\"match_{{template}}\": {}}\"",
            "template_vars" : {
                "template" : "all"
            }
        }
    }
}

You register a template by storing it in the conf/scripts directory of
elasticsearch. In order to execute the stored template reference it in the query parameters:

GET _search
{
    "query": {
        "template": {
            "template_string": "storedTemplate",
            "template_vars" : {
                "template" : "all"
            }
        }
    }
}

Template language

Templating is based on Mustache. Substitution of tokens works as follows:

            "template_string": "{\"match_{{template}}\": {}}\"",
            "template_vars" : {
                "template" : "all"
@kimchy
Copy link
Member

kimchy commented Jan 24, 2014

can we also support

{
    "query": {
        "template": {
            "query": {
                "match_{{template}}" : {}
            }
            "params" : {
                "template" : "all"
            }
        }
    }
}

this is much more readable? we can also make it super fast by using a new XContentBuilder and calling copyCurrentStructure, and running the template on it.

UPDATE:

Also, based on above, I would go with

{
    "query": {
        "template": {
            "query": "{\"match_{{template}}\": {}}\"",
            "params" : {
                "template" : "all"
            }
        }
    }
}

when in "string/bytes" mode, we can detect that, and its more readable.

@uboness
Copy link
Contributor

uboness commented Jan 24, 2014

or

{
    "query": {
        "template": {
            "template": {
                "match_{{template}}" : {}
            }
            "vars" : {
                "template" : "all"
            }
        }
    }
}

@uboness
Copy link
Contributor

uboness commented Jan 24, 2014

and

{
    "query": {
        "template": {
            "name": "storedTemplate",
            "vars" : {
                "template" : "all"
            }
        }
    }
}

@MaineC
Copy link
Author

MaineC commented Jan 24, 2014

Makes sense - makes it also easier to simply copy/paste existing queries into templates.

@ghost ghost assigned MaineC Jan 24, 2014
@MaineC MaineC mentioned this issue Feb 6, 2014
Merged
MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Feb 14, 2014
Add mustache templating to query execution.
6725fc8 
Adds support for storing mustache based query templates that can later be filled
with query parameter values at execution time.

Relates to elastic#4879
MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Feb 14, 2014
Rename template query parameter names.
e1806a9 
This makes the names of the parameters a template query accepts shorter and
clearer.

Relates to elastic#4879
MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Feb 14, 2014
Refactoring to allow for non quoted templates.
21903db 
As per suggestion in elastic#4879 modified the implementation to allow for templates
that are not quoted but legitimate json. Leaving the option to completely quote
in - needed anyway for support for referencing templates stored in the config
directory.

Relates to elastic#4879
MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Feb 14, 2014
Mark mustache as optional dependency.
4b726eb 
Relates to elastic#4879
MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Feb 14, 2014
Fixing pull request comments by s1monw.
34a470f 
Relates to elastic#4879
MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Feb 14, 2014
Use Elasticsearch provided assertions.
5abf744 
Instead of digging out the number of hits use Elasticsearch provided helper
method to check the hit count. (Thanks @spinscale for the suggestion)

In addition simplify the test slightly by factoring index setup into
separate method executed before test execution.

Relates to elastic#4879
@MaineC MaineC mentioned this issue Feb 14, 2014
Closed
MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Feb 17, 2014
Fix more code review comments.
a2d1ba3 
Relates to elastic#4879
MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Feb 17, 2014
Missed a dependency of mustache in lib folder
be48f5f 
When first moving mustache from shaded jar to lib dir the dependency to guava
was missing...

Relates to elastic#4879
MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Feb 19, 2014
Final cleanup work.
3412f46 
Relates to elastic#4879

This moves mustache back to being shaded - having guava (which is a needed
dependency of mustache) explicitly in the distribution files isn't an option:
Used by too many projects and thus likely to cause version conflicts downstream.

In addition this adds a rest test for the query templating feature.
MaineC pushed a commit that referenced this issue Feb 20, 2014
Add mustache templating to query execution.
e3008be 
Adds support for storing mustache based query templates that can later be filled
with query parameter values at execution time. Templates may be both quoted,
non-quoted and referencing templates stored in config/scripts/*.mustache by file
name.

See docs/reference/query-dsl/queries/template-query.asciidoc for templating
examples.

Implementation detail: mustache itself is being shaded as it depends directly on
guava - so having it marked optional but included in the final distribution
raises chances of version conflicts downstream.

Fixes #4879
@MaineC MaineC closed this as completed in 48004ff Feb 20, 2014
MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Apr 2, 2014
Add some more documentation.
34c8810 
Relates to elastic#4879
MaineC pushed a commit that referenced this issue Apr 24, 2014
@javanna
Add some more documentation to TemplateQueryParser
dd75f3a 
Relates to #4879
MaineC pushed a commit that referenced this issue Apr 24, 2014
@javanna
Add some more documentation to TemplateQueryParser
dcfc7ce 
Relates to #4879
@gedl
Copy link

gedl commented Jul 10, 2014

Is it possible to create templates at runtime as opposed to pre-package them in the config folder ?

@clintongormley
Copy link

@gedl yes - see template query and search template

@MaineC MaineC removed their assignment Mar 10, 2015
@clintongormley clintongormley added :Search/Search Search-related issues that do not fall into other categories and removed :Query DSL labels Feb 14, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>feature :Search/Search Search-related issues that do not fall into other categories v1.1.0 v2.0.0-beta1
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@kimchy @clintongormley @MaineC @uboness @spinscale @gedl