Remove experimental event.original definition

[ebeahan]

Summary

While testing the experimental build artifacts, I discovered the generated experimental ES index templates will not install.

Error Details

Elasticsearch returns a HTTP 400 error when attempting to install the generated ES index template:

 curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_template/test -d @experimental/generated/elasticsearch/7/template.json 
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"The field [original] cannot have index = false"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: The field [original] cannot have index = false","caused_by":{"type":"mapper_parsing_exception","reason":"The field [original] cannot have index = false"}},"status":400}

Proposed Solution

Set index: true in the experimental schema for event.original to override. Also correct in the stage 2 wildcard RFC field definitions.

UPDATED: Agreement reached in #1015 to remove event.original from the initial wildcard field migration. Updating this PR to remove the field from the experimental definitions for 1.7 to avoid users encountering this issue.

[ebeahan]

I think this makes sense to backport for 1.7.

[webmat]

If we move forward with this PR, I 100% agree we should backport to 1.7.

However @leehinman had pointed out in #970 (comment) that indexing it could lead to a significant performance hit. So we kept the wildcard type and decided to leave it index: false.

This may be an inconsistency in how Elasticsearch deals with type:keyword + index:false vs doing the same with type:wildcard.

I think a more appropriate fix would be to adjust our ES template generator, and not set a type at all, when we set index:false.

[ebeahan]

However @leehinman had pointed out in #970 (comment) that indexing it could lead to a significant performance hit. So we kept the wildcard type and decided to leave it index: false.

That's right - we did make the decision to remove. Apologies for not recalling that point.

[ebeahan]

Compared the result of setting index: false on the three keyword family field types.

keyword

request:

PUT keyword-index
{
  "mappings": {
    "properties": {
      "keyword": {
        "type": "keyword",
        "index": false
      }
    }
  }
}

response:

{
  "acknowledged" : true
}

constant_keyword

request:

{
  "mappings": {
    "properties": {
      "constant_keyword": {
        "type": "constant_keyword",
        "index": false
      }
    }
  }
}

response:

{
  "acknowledged" : true
}

wildcard

request:

PUT wildcard-index
{
  "mappings": {
    "properties": {
      "wildcard": {
        "type": "wildcard",
        "index": false
      }
    }
  }
}

response:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "The field [wildcard] cannot have index = false"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "Failed to parse mapping [_doc]: The field [wildcard] cannot have index = false",
    "caused_by" : {
      "type" : "mapper_parsing_exception",
      "reason" : "The field [wildcard] cannot have index = false"
    }
  },
  "status" : 400
}

@markharwood - We came across this inconsistency when using type: wildcard + index: false compared to the other keyword family types. Would you be able to comment if this expected for wildcard?

[ebeahan]

I think a more appropriate fix would be to adjust our ES template generator, and not set a type at all, when we set index:false

I experimented with omitting the type but saw exceptions returned when trying to create the index:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "No type specified for field [none]"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "Failed to parse mapping [_doc]: No type specified for field [none]",
    "caused_by" : {
      "type" : "mapper_parsing_exception",
      "reason" : "No type specified for field [none]"
    }
  },
  "status" : 400
}

Is there a proper way to omit the type?

[ebeahan]

Confirmed wildcard doesn't support setting index: false (or doc_values: false) by design. Reference: https://github.com/elastic/elasticsearch/pull/49993/files#r357572257.

Will move discussion to #1015 for determining how to proceed with handling event.original and wildcard.

[webmat]

Alright, if Travis is happy with it, LGTM!