Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ES dynamic template functionality is broken in Graylog 1.3 #1624

Closed
jehuty0shift opened this issue Dec 10, 2015 · 3 comments
Closed

ES dynamic template functionality is broken in Graylog 1.3 #1624

jehuty0shift opened this issue Dec 10, 2015 · 3 comments
Assignees
Milestone

Comments

@jehuty0shift
Copy link

@jehuty0shift jehuty0shift commented Dec 10, 2015

Hello,

Since Graylog 1.3, the use of clever dynamic template to specify values type and doc_values is not possible anymore. Since you create the index with your own index template, you prevent anyone to specify any special mapping with a dynamic template.
Here is the mapping we have now with graylog 1.3

{
  "graylog2_321" : {
    "mappings" : {
      "message" : {
        "dynamic_templates" : [ {
          "internal_fields" : {
            "mapping" : {
              "index" : "not_analyzed",
              "doc_values" : true
            },
            "match" : "gl2_*"
          }
        }, {
          "store_generic" : {
            "mapping" : {
              "index" : "not_analyzed"
            },
            "match" : "*"
          }
        },  {
          "num_suffix" : {
            "mapping" : {
              "type" : "double",
              "doc_values" : true
            },
            "match" : "*_num"
          }
        }, {
          "others" : {
            "mapping" : {
              "index" : "not_analyzed",
              "type" : "string",
              "doc_values" : true
            },
            "match" : "*"
          }
...
"properties" : {
          "thread" : {
            "type" : "long"
          },
...

"thread" should here a string, but it is now a long. Note that also doc_values can not be specified anymore for non analyzed fields.
This is a problem since it was useful to prevent collisions to happen by enforcing conventions with ES.
I have also seen the dynamic_templates features as a solution proposed in some issues. I changed the order of the template but it doesn't help. It is working perfectly fine in 1.2 since the dynamic template mapping of Graylog is applied after the dynamic template present in ES.

@joschi
Copy link
Contributor

@joschi joschi commented Dec 10, 2015

Refs #1502.

@bernd bernd self-assigned this Dec 11, 2015
@bernd bernd added the bug label Dec 11, 2015
@bernd
Copy link
Member

@bernd bernd commented Dec 11, 2015

@jehuty0shift Thank you for the report. We are able to reproduce the issue and are working on a fix.

bernd added a commit that referenced this issue Dec 11, 2015
…ation

In commit edf0d8b we changed the index creation process from "create
index" + "install index mapping" to one operation to avoid race
conditions.

This broke setups where users had installed custom Elasticsearch index
templates because the user templates where applied after our mapping was
installed into the index. Before edf0d8b the user templates have been
applied before we installed our mapping because the index creation was
split into two steps.

We now install a "graylog-internal" index template into Elasticsearch
with a "order" or -99 to make sure user templates are able to override
our settings.

Fixes #1624
bernd added a commit that referenced this issue Dec 14, 2015
…ation

In commit edf0d8b we changed the index creation process from "create
index" + "install index mapping" to one operation to avoid race
conditions.

This broke setups where users had installed custom Elasticsearch index
templates because the user templates where applied after our mapping was
installed into the index. Before edf0d8b the user templates have been
applied before we installed our mapping because the index creation was
split into two steps.

We now install a "graylog-internal" index template into Elasticsearch
with a "order" or -99 to make sure user templates are able to override
our settings.

Fixes #1624
@joschi joschi added this to the 1.3.1 milestone Dec 15, 2015
@joschi
Copy link
Contributor

@joschi joschi commented Dec 15, 2015

Fixed in #1628.

@joschi joschi closed this Dec 15, 2015
joschi pushed a commit that referenced this issue Dec 17, 2015
joschi pushed a commit that referenced this issue Dec 17, 2015
Refs #1624, refs #1628
(cherry picked from commit a826c05)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants