Skip to content

Commit

Permalink
[ML] Reduce warning logging from get categories Grok pattern creation (
Browse files Browse the repository at this point in the history
…#73373)

The Grok pattern creator used within the get categories action
works by looking at the examples for each category. Sometimes
these examples are truncated, and cannot be used for Grok pattern
determination. Previously when this happened we would log a warning,
but that caused a feedback loop in the case where the Elasticsearch
logs themselves were being categorized: the warning messages would
end up creating new categories in the Elasticsearch log categorization,
with very long examples that would be truncated, leading to the same
problem occurring yet again.

This change reduces the warning log to trace for truncated log
messages, and also removes the example from the log message to
shorten it.
  • Loading branch information
droberts195 committed May 26, 2021
1 parent 117f62a commit 216b30a
Showing 1 changed file with 7 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.message.ParameterizedMessage;
import org.elasticsearch.grok.Grok;

import java.util.ArrayList;
Expand Down Expand Up @@ -124,7 +125,12 @@ public static String findBestGrokMatchFromExamples(String jobId, String regex, C
// the message was very long, and the example was truncated. In this
// case we will have appended an ellipsis to indicate truncation.
assert example.endsWith("...") : exampleProcessor.pattern() + " did not match non-truncated example " + example;
logger.warn("[{}] Pattern [{}] did not match example [{}]", jobId, exampleProcessor.pattern(), example);
if (example.endsWith("...")) {
logger.trace(() -> new ParameterizedMessage("[{}] Pattern [{}] did not match truncated example",
jobId, exampleProcessor.pattern()));
} else {
logger.warn("[{}] Pattern [{}] did not match non-truncated example [{}]", jobId, exampleProcessor.pattern(), example);
}
}
}

Expand Down

0 comments on commit 216b30a

Please sign in to comment.