Performance & Troubleshooting

Expert guidance on optimizing the EventProcessor and solving common implementation issues.

Performance Optimization

Data Type Narrowing: Evaluated rules only for the data types they are built for.
Efficient CEL: Avoid large array lookups or deep nested loops in where clauses.
Correlation Windows: Use the smallest possible within window that fulfills the logic.
Limit Correlation: Searching millions of logs is expensive. Filter as much as possible using with fields.

Early Exit: Use where clauses to skip unnecessary steps.
JSON over Regex: If a log is JSON, always use the json step instead of grok. It is significantly faster.
Field Cleanup: Remove temporary log.* fields when no longer needed. Large, redundant strings increase indexing latency and disk usage. Note that the raw field is preserved for audit and cannot be removed by parsing steps.

CPU: Parsing and CEL evaluation are CPU-intensive. Add cores for higher EPS (Events Per Second).
RAM: OpenSearch and the EventProcessor cache rely on system memory.
I/O: Use SSDs for OpenSearch data directories to avoid bottlenecks during high-volume correlation searches.

Symptom	Potential Cause	Solution
Rule not triggering	Mismatched `dataType` or `where` logic.	Check the "Log Explorer" to ensure the event has the correct type and all fields referenced in the rule.
Filter not parsing	Incorrect `source` field or syntax error.	Validate that the `source` field (e.g., `raw`) exists in the input. Check processor logs for YAML parsing errors.
Alert fatigue	Missing `deduplicateBy`.	Group alerts by `origin.ip` or `origin.user` to consolidate findings.
High Latency	Complex `grok` or large `correlation` windows.	Optimize regex patterns or reduce the `within` time window.

Logs: Check the EventProcessor logs in your deployment (container logs or configured log path).
Validator: Use a YAML linter on your rules/filters.
Trace: Use an event generator to send a single test log and track it through the system.