Add .caseless subfield to process.name & process.executable

[w0rk3r]

Summary

Related PR: elastic/integrations#9850

This PR aims to add a subfield to the process.name and process.executable fields to improve the compatibility of data sources like System, Sysmon, etc., with our Elastic Defend data. This enables us to handle language limitations in KQL more effectively.

Elastic Defend Mapping:

[efd6]

It's a pity the can doesn't describe the contents; in a perfect world calling this "lowercase" would have been better, but we are where we are.

Please ensure that the final commit message is not just a GitHub fusion of the commit messages here.

[andrewkroh]

I was originally reluctant about the naming because I felt that .lowercase is more meaningful for what is stored. But the other aspect of the normalizer is the search-time normalization, and this actively lowercases the search term making the query case-insensitive (or "caseless"). So I think I have rationalized the naming for myself, and I am OK with .caseless.

But I would like to hear opinions from others on where they stand with the naming. I have tagged a few teams, but anyone is welcome to chime in.

[felixbarny]

Looks like this has mainly been added for security use cases and because of language limitations in KQL. Have we considered enhancing KQL, moving to ES|QL, or only adding the new sub-fields to specific integrations or in security project types?
My concern is the storage impact for observability use cases.

[eyalkoren]

@w0rk3r @andrewkroh @efd6 we need your input on @felixbarny's question above and on @ruflin's question, so we can move on with elastic/elasticsearch#110740. In the mean time, ECS is not fully covered by our own mappings, which are widely used, and we have these tests failing daily.

A bit of context - ecs@mappings are dynamic mappings that fully cover ECS. In order to verify that they are kept up-to-date with ECS, we employ daily tests that fail unless all fiedls are properly covered. Since you merged this PR, ecs@mappings is out of sync with ECS.
Please respond here and in the related fix.
Thanks!

[andrewkroh]

I pushed for standardizing this multi-field because I think it is a negative user-experience to have varying mapping definitions for an ECS field across integrations. I think users expect consistency when dealing with ECS. i.e. No matter were field X is used, it has the same semantics and characteristics.

I do share the storage concerns even for security use cases. So if there something that can be done on the query-side to obviate the need for this multi-field that would be a great.

@elastic/threat-research-and-detection-engineering team, is it feasible to transition to ES|QL for the queries that depend on the "caseless" multi-fields?

@felixbarny do you have any insight into whether KQL could support this case-insensitive query?

If either ES|QL or KQL are probable solutions then I would be OK with temporary adding an inconsistent definition to the integrations that need the .caseless multi-field to bridge the gap and we can revert this PR (which has not be released).

[ebeahan]

do you have any insight into whether KQL could support this case-insensitive query?

Linking a couple of past discussions: elastic/kibana#80591, elastic/kibana#55378

[w0rk3r]

@elastic/threat-research-and-detection-engineering team, is it feasible to transition to ES|QL for the queries that depend on the "caseless" multi-fields?

Not at the moment, mainly because new_terms rules don't support any other language than KQL. We avoid KQL whenever possible, but we still have limitations and performance scenarios that force us to use it. Also, we support kibana versions where ESQL isn't available or is running previous versions (which can lead to bugs).

But this isn't just for our prebuilt content. Some customers using Sysmon or Windows security logs have custom processors to add a lowercase version of these fields (we saw this in the last Locked Shields we participated), and others don't know they are required to do this and end up having false negatives (in our prebuilt content, custom use cases, and rules imported from Sigma).

I'm ok with going the integrations route again if needed.

[felixbarny]

I think users expect consistency when dealing with ECS. i.e. No matter were field X is used, it has the same semantics and characteristics

That makes sense. However, in this case, it seems to me that we're adding a subfield only to work around limitations in KQL and the fact that some rules don't support any other language than KQL. To me, this indicates that these sub-fields shouldn't be part of the default set of fields for ECS.

Am I right to assume that our preferred long-term solution is the rely on ES|QL case-insensitive behavior? Or are there other strong reasons, like a more efficient search, that would warrant the increased storage impact for all use cases?

If the long-term solution is ES|QL, it seems like we shouldn't add .caseless fields to ECS. To unblock us in the short-term, integration-specific sub-fields seem more appropriate.

[w0rk3r]

I just opened this PR in integrations to add them to the specific integrations if that is the way we choose to proceed: elastic/integrations#10533

[ruflin]

++ on keeping the change to a minimum at first. Should we revert this ECS change?

[andrewkroh]

Revert is complete. See #2350.

[nicpenning]

One of the biggest compliants I hear about Kibana/Elastic is this specific issue. Knowing when to use the right case sensitivity. Users have to take extra steps continually to attempt upper/lower or wild card searches to find what they ate looking for. That is why we force many fields to lowercase so they don't have to think about this. KQL is still 99% of the searches because it is so simple and it gets the job done. ES|QL is not a long term solution IMO. Giving KQL the ability to be caseless in other ways would be great if the mappings won't do it. Host names is a other area where this occurs and unfortunately system indices / areas of the stack this problematic because things like OSQuery require exact matches but Fleet seems to work okay. Perhaps optional multifield types in ECS could allow the best of both worlds where there is no issue/conflict with having all or just 1 extra multifield. Something that probably doesn't exist as a capability today but just a thought. It's a little uneasy that the MDE integration just got this treatment which drifts further away from ECS if I am tracking this thread right. If you ask me, the fields should be simply lowercased today and retain the original process.name in integration_name.process.name.original. It is important to see the case sensitivity because that could add valuable insight into threat activity.

Just my 2 cents.