Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use wildcard in indices for searching #1232

Merged
merged 1 commit into from May 6, 2019

Conversation

Projects
None yet
3 participants
@pwnbus
Copy link
Collaborator

commented Apr 24, 2019

Now that ES is more efficient with timestamped data, we can expand the search indices across all events-, and all alerts-. Previously, we were restricting our search window between 24 - 48 hours (events to events-previously).

We ran some benchmarks, and it turns out there isn't a very large performance hit when expanding to wildcards:

------------ benchmark: 2 tests -----------ons (2/2)
Name (time in ms)            Mean          
-------------------------------------------
test_events_weekly        90.7345 (1.0)    
test_events_wildcard     105.5318 (1.16)   
-------------------------------------------
@pwnbus

This comment has been minimized.

Copy link
Collaborator Author

commented Apr 24, 2019

We'll have to release a new version of mozdef_util as we changed the default indices.

@jeffbryner

This comment has been minimized.

Copy link
Collaborator

commented Apr 24, 2019

Any impact to memory utilization?

@Phrozyn

This comment has been minimized.

Copy link
Contributor

commented Apr 25, 2019

We should take a look at that. JVM usage could be increased by this method. Good call, I didn't think of that.

@pwnbus

This comment has been minimized.

Copy link
Collaborator Author

commented May 2, 2019

I took a look at CPU, memory (via top) and the JVM during the testing and it didn't seem to have any noticeable difference.

@jeffbryner

This comment has been minimized.

Copy link
Collaborator

commented May 2, 2019

I took a look at CPU, memory (via top) and the JVM during the testing and it didn't seem to have any noticeable difference.

OK, it's likely they've gotten much better at only sending queries around to places that are likely to have hits + JVM would increase for hits rather than misses.

@Phrozyn

Phrozyn approved these changes May 2, 2019

@pwnbus pwnbus merged commit 8a0df3a into master May 6, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@pwnbus pwnbus deleted the use_index_wildcards branch May 6, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.