[Discuss] ignore_above on flattened fields

[ebeahan]

Question for discussion: Should ECS set ignore_above by default on flattened fields?

Context

Pointed out in #2237, ECS doesn't yet support setting ignore_above explicitly set on flattened fields (fix incoming).

As a convention, the artifacts generated by ECS set the ignore_above parameter on all keyword fields. Setting ignore_above avoids exceptions due to Lucene size limit for a single term and avoids unnecessarily indexing longer string values.

Other considerations

• If ECS did set ignore_above on flattened type fields, should ECS use the existing convention of ignore_above: 1024?
• Implications
  • If set, any fields with character counter > ignore_above will no longer be indexed and only appear in _source.
  • On the other hand, any flattened field values larger than the Lucene limit of 32766 bytes with cause an exception. Unsure if this affects the entire flattened field or just the specific leaf field that's oversized (but could be easily tested).