Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for executing queries based on pre-defined templates #4879

Closed
MaineC opened this issue Jan 24, 2014 · 6 comments

Comments

Projects
None yet
6 participants
@MaineC
Copy link
Contributor

commented Jan 24, 2014

It would be nice to be able to store pre-defined query templates that can be referenced and filled with parameter values at query time. This can be in particular useful to be able to quickly replay queries with slightly different templates but identical parameter values e.g. in order to compare slightly different ways to formulate Elasticsearch queries wrt. runtime performance and actual results returned.

For developing and testing the template, both, template_string and template_vars can be submitted as part of the search request:

GET _search
{
    "query": {
        "template": {
            "template_string": "{\"match_{{template}}\": {}}\"",
            "template_vars" : {
                "template" : "all"
            }
        }
    }
}

You register a template by storing it in the conf/scripts directory of
elasticsearch. In order to execute the stored template reference it in the query parameters:

GET _search
{
    "query": {
        "template": {
            "template_string": "storedTemplate",
            "template_vars" : {
                "template" : "all"
            }
        }
    }
}

Template language

Templating is based on Mustache. Substitution of tokens works as follows:

            "template_string": "{\"match_{{template}}\": {}}\"",
            "template_vars" : {
                "template" : "all"
@kimchy

This comment has been minimized.

Copy link
Member

commented Jan 24, 2014

can we also support

{
    "query": {
        "template": {
            "query": {
                "match_{{template}}" : {}
            }
            "params" : {
                "template" : "all"
            }
        }
    }
}

this is much more readable? we can also make it super fast by using a new XContentBuilder and calling copyCurrentStructure, and running the template on it.

UPDATE:

Also, based on above, I would go with

{
    "query": {
        "template": {
            "query": "{\"match_{{template}}\": {}}\"",
            "params" : {
                "template" : "all"
            }
        }
    }
}

when in "string/bytes" mode, we can detect that, and its more readable.

@uboness

This comment has been minimized.

Copy link
Contributor

commented Jan 24, 2014

or

{
    "query": {
        "template": {
            "template": {
                "match_{{template}}" : {}
            }
            "vars" : {
                "template" : "all"
            }
        }
    }
}
@uboness

This comment has been minimized.

Copy link
Contributor

commented Jan 24, 2014

and

{
    "query": {
        "template": {
            "name": "storedTemplate",
            "vars" : {
                "template" : "all"
            }
        }
    }
}
@MaineC

This comment has been minimized.

Copy link
Contributor Author

commented Jan 24, 2014

Makes sense - makes it also easier to simply copy/paste existing queries into templates.

@ghost ghost assigned MaineC Jan 24, 2014

@MaineC MaineC referenced this issue Feb 6, 2014

Merged

Feature/4879 #5036

MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Feb 14, 2014

Isabel Drost-Fromm
Add mustache templating to query execution.
Adds support for storing mustache based query templates that can later be filled
with query parameter values at execution time.

Relates to elastic#4879

MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Feb 14, 2014

Isabel Drost-Fromm
Rename template query parameter names.
This makes the names of the parameters a template query accepts shorter and
clearer.

Relates to elastic#4879

MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Feb 14, 2014

Isabel Drost-Fromm
Refactoring to allow for non quoted templates.
As per suggestion in elastic#4879 modified the implementation to allow for templates
that are not quoted but legitimate json. Leaving the option to completely quote
in - needed anyway for support for referencing templates stored in the config
directory.

Relates to elastic#4879

MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Feb 14, 2014

MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Feb 14, 2014

MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Feb 14, 2014

Isabel Drost-Fromm
Use Elasticsearch provided assertions.
Instead of digging out the number of hits use Elasticsearch provided helper
method to check the hit count. (Thanks @spinscale for the suggestion)

In addition simplify the test slightly by factoring index setup into
separate method executed before test execution.

Relates to elastic#4879

MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Feb 17, 2014

MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Feb 17, 2014

Isabel Drost-Fromm
Missed a dependency of mustache in lib folder
When first moving mustache from shaded jar to lib dir the dependency to guava
was missing...

Relates to elastic#4879

MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Feb 19, 2014

Isabel Drost-Fromm
Final cleanup work.
Relates to elastic#4879

This moves mustache back to being shaded - having guava (which is a needed
dependency of mustache) explicitly in the distribution files isn't an option:
Used by too many projects and thus likely to cause version conflicts downstream.

In addition this adds a rest test for the query templating feature.

MaineC pushed a commit that referenced this issue Feb 20, 2014

Isabel Drost-Fromm
Add mustache templating to query execution.
Adds support for storing mustache based query templates that can later be filled
with query parameter values at execution time. Templates may be both quoted,
non-quoted and referencing templates stored in config/scripts/*.mustache by file
name.

See docs/reference/query-dsl/queries/template-query.asciidoc for templating
examples.

Implementation detail: mustache itself is being shaded as it depends directly on
guava - so having it marked optional but included in the final distribution
raises chances of version conflicts downstream.

Fixes #4879

@MaineC MaineC closed this in 48004ff Feb 20, 2014

MaineC pushed a commit to MaineC/elasticsearch that referenced this issue Apr 2, 2014

MaineC pushed a commit that referenced this issue Apr 24, 2014

MaineC pushed a commit that referenced this issue Apr 24, 2014

@gedl

This comment has been minimized.

Copy link

commented Jul 10, 2014

Is it possible to create templates at runtime as opposed to pre-package them in the config folder ?

@clintongormley

This comment has been minimized.

Copy link
Member

commented Jul 11, 2014

@MaineC MaineC removed their assignment Mar 10, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.