Adopting Wildcard Fields

[jamiehynds]

Wildcard is a data type for Elasticsearch string fields introduced in Elasticsearch 7.9. Wildcard optimizes performance for queries using wildcards (*) and regex, allowing users to perform grep-like searches without the limitations of the existing text and keyword types.

ECS is supporting wildcards (RFC here) for the following fields:

Field Set	Field(s)
error	error.stack_trace
http	http.request.body.content http.response.body.content
process	process.command_line
registry	registry.data.strings
url	url.full url.original url.path

There is an impact on indexing throughput (5% decrease on average) and storage (5% increase on average) based on our performance testing of wildcard fields.

With a set of security integrations GA'ing in 7.14, we will update these integrations to change keyword fields to wildcard in 7.14. Beats modules will not be updated to ensure existing Beats users are not impacted.

Integrations to be updated for 7.14

• Windows Security (System integration) [system] Use wildcard type for security stream #1185
• Windows PowerShell [windows] Use wildcard field type #1161
• Windows Sysmon [windows] Use wildcard field type #1161
  - [ ] auditd
• Suricata [suricata] Use wildcard type #1162
• Zeek [zeek] Use wildcard type #1164
• Netflow [netflow] Use wildcard type #1179
• O365 [o365] Use wildcard type #1180
• Cisco [cisco] Use wildcard type #1181
• Palo Alto [panw] Use wildcard type #1182
  - [ ] Okta
• Fortinet [fortinet] Use wildcard type #1183

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[mtojek]

While working on the elastic/package-spec#63 I introduced a simple dependency management to import ECS field definitions (name, type, description) directly from ECS repository. I will try to enable this feature for some packages in elastic/integrations.

For reference: https://github.com/elastic/elastic-package/blob/master/test/packages/nginx/data_stream/access/fields/ecs.yml#L6

Sample PR: #1171

Is it something that will be useful here?