[crowdstrike] Expose FDR cache options for more flexibility #9063
Conversation
marc-gr
added
enhancement
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
marc-gr
force-pushed
the
improve_crowdstrike_cache_options
branch
from
b40269e
to
f195c59
Compare
🚀 Benchmarks reportTo see the full report comment with |
| - name: metadata_cache_capacity | ||
| required: true | ||
| show_user: true | ||
| title: Metadata cache capacity | ||
| description: The maximum amount of metadata objects to cache. Operations that would cause the capacity to be exceeded will result in evictions of the oldest elements. The capacity should not be lower than the number of elements that are expected to be referenced when processing the input as evicted elements are lost. Values at or below zero indicate no limit. | ||
| type: text | ||
| multi: false | ||
| default: 0 |
There was a problem hiding this comment.
I'm not sure it's a good idea to expose this. If it is exposed, the FOOTGUN sign needs to be a lot bigger.
There was a problem hiding this comment.
As in, unexpectedly evicting objects, or?
There was a problem hiding this comment.
Yes. If the user sets this too low, they will have holes in their enrichment.
There was a problem hiding this comment.
Wouldn't this also happen after the TTL expires?
There was a problem hiding this comment.
All to puts will call the store's evictExpired method. This will do TTL-base expiry, but if the capacity of the store is positive there will be enforced eviction tighter than the TTL until the number of elements is within capacity. This means that earlier-seen elements will be absent from the cache and so we'll have an enrichment blind spot.
The logic for this is here https://github.com/elastic/beats/blob/7546ae1ceefa19c9a4886ab012fa661eaeaccf4b/libbeat/processors/cache/mem_store.go#L174
But overall, yes, but we have a clear understanding of the dynamics of TTL; we can know definitively how long the refresh cycle is because it is configured, but the numbers of entities in the aidmaster/user_info are a non-configured function of the system that is being monitored. In principle they can be known and accounted for, but in practice, I can see situation where the integration is configured in one setting, the system gets bigger and then metadata is not applied in confusing ways. I'd prefer to reduce the likelihood of that.
|
Just a reminder user enrichment doesn't work #8742 |
|
@efd6 I added a warning to the capacity setting and set it hidden by default. If you still think there is no reason to expose it in any context, I am not against removing it. |
💚 Build Succeeded
History
|
|
0.0% Duplication on New Code
|
Package crowdstrike - 1.29.0 containing this change is available at https://epr.elastic.co/search?package=crowdstrike |
Proposed commit message
Exposes cache options for the FDR data stream.
Using the cache leads to more memory consumption, by exposing this users can tweak how cache is evicted to limit memory usage if needed.
Checklist
changelog.ymlfile.