New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ES dynamic template functionality is broken in Graylog 1.3 #1624

Closed
jehuty0shift opened this Issue Dec 10, 2015 · 3 comments

Comments

Projects
None yet
3 participants
@jehuty0shift

jehuty0shift commented Dec 10, 2015

Hello,

Since Graylog 1.3, the use of clever dynamic template to specify values type and doc_values is not possible anymore. Since you create the index with your own index template, you prevent anyone to specify any special mapping with a dynamic template.
Here is the mapping we have now with graylog 1.3

{
  "graylog2_321" : {
    "mappings" : {
      "message" : {
        "dynamic_templates" : [ {
          "internal_fields" : {
            "mapping" : {
              "index" : "not_analyzed",
              "doc_values" : true
            },
            "match" : "gl2_*"
          }
        }, {
          "store_generic" : {
            "mapping" : {
              "index" : "not_analyzed"
            },
            "match" : "*"
          }
        },  {
          "num_suffix" : {
            "mapping" : {
              "type" : "double",
              "doc_values" : true
            },
            "match" : "*_num"
          }
        }, {
          "others" : {
            "mapping" : {
              "index" : "not_analyzed",
              "type" : "string",
              "doc_values" : true
            },
            "match" : "*"
          }
...
"properties" : {
          "thread" : {
            "type" : "long"
          },
...

"thread" should here a string, but it is now a long. Note that also doc_values can not be specified anymore for non analyzed fields.
This is a problem since it was useful to prevent collisions to happen by enforcing conventions with ES.
I have also seen the dynamic_templates features as a solution proposed in some issues. I changed the order of the template but it doesn't help. It is working perfectly fine in 1.2 since the dynamic template mapping of Graylog is applied after the dynamic template present in ES.

@joschi

This comment has been minimized.

Contributor

joschi commented Dec 10, 2015

Refs #1502.

@bernd bernd self-assigned this Dec 11, 2015

@bernd bernd added the bug label Dec 11, 2015

@bernd

This comment has been minimized.

Member

bernd commented Dec 11, 2015

@jehuty0shift Thank you for the report. We are able to reproduce the issue and are working on a fix.

bernd added a commit that referenced this issue Dec 11, 2015

Install a Graylog index template instead of set mappings on index cre…
…ation

In commit edf0d8b we changed the index creation process from "create
index" + "install index mapping" to one operation to avoid race
conditions.

This broke setups where users had installed custom Elasticsearch index
templates because the user templates where applied after our mapping was
installed into the index. Before edf0d8b the user templates have been
applied before we installed our mapping because the index creation was
split into two steps.

We now install a "graylog-internal" index template into Elasticsearch
with a "order" or -99 to make sure user templates are able to override
our settings.

Fixes #1624

bernd added a commit that referenced this issue Dec 14, 2015

Install a Graylog index template instead of set mappings on index cre…
…ation

In commit edf0d8b we changed the index creation process from "create
index" + "install index mapping" to one operation to avoid race
conditions.

This broke setups where users had installed custom Elasticsearch index
templates because the user templates where applied after our mapping was
installed into the index. Before edf0d8b the user templates have been
applied before we installed our mapping because the index creation was
split into two steps.

We now install a "graylog-internal" index template into Elasticsearch
with a "order" or -99 to make sure user templates are able to override
our settings.

Fixes #1624

@joschi joschi added this to the 1.3.1 milestone Dec 15, 2015

@joschi

This comment has been minimized.

Contributor

joschi commented Dec 15, 2015

Fixed in #1628.

@joschi joschi closed this Dec 15, 2015

joschi added a commit that referenced this issue Dec 17, 2015

joschi added a commit that referenced this issue Dec 17, 2015

Make Elasticsearch index template name configurable
Refs #1624, refs #1628
(cherry picked from commit a826c05)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment