Skip to content

Commit

Permalink
[ML] Do not write model size stats after autodetect no-op
Browse files Browse the repository at this point in the history
Similar to the fix for state and quantiles in elastic#437,
if no input is received and time is not advanced then
there is no need to write model size stats when the
autodetect process exits. Doing this can actually
cause a problem for a job that has never ever seen
any input, as the unnecessary model size stats were
written with a negative timestamp. This change also
adds an extra defensive check to prevent that ever
happening, although the only situation when it is
thought to be possible should be prevented by the
first change.

Backport of elastic#512
  • Loading branch information
droberts195 committed Jun 27, 2019
1 parent f0ae0b2 commit ad0b97e
Show file tree
Hide file tree
Showing 2 changed files with 17 additions and 5 deletions.
6 changes: 6 additions & 0 deletions docs/CHANGELOG.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,12 @@ to the model. (See {ml-pull}214[#214].)
== {es} version 7.0.0-alpha1
== {es} version 6.8.2
=== Bug Fixes
* Don't write model size stats when job is closed without any input {ml-pull}512[#512] (issue: {ml-issue}394[#394])
== {es} version 6.7.2
=== Enhancements
Expand Down
16 changes: 11 additions & 5 deletions lib/api/CAnomalyJob.cc
Original file line number Diff line number Diff line change
Expand Up @@ -221,14 +221,14 @@ bool CAnomalyJob::handleRecord(const TStrStrUMap& dataRowFields) {

void CAnomalyJob::finalise() {
// Persist final state of normalizer iff an input record has been handled or time has been advanced.
if (this->isPersistenceNeeded("quantiles state")) {
if (this->isPersistenceNeeded("quantiles state and model size stats")) {
m_JsonOutputWriter.persistNormalizer(m_Normalizer, m_LastNormalizerPersistTime);
}

// Prune the models so that the final persisted state is as neat as possible
this->pruneAllModels();
// Prune the models so that the final persisted state is as neat as possible
this->pruneAllModels();

this->refreshMemoryAndReport();
this->refreshMemoryAndReport();
}

// Wait for any ongoing periodic persist to complete, so that the data adder
// is not used by both a periodic periodic persist and final persist at the
Expand Down Expand Up @@ -1267,6 +1267,12 @@ void CAnomalyJob::writeOutModelPlot(const TModelPlotDataVec& modelPlotData) {
}

void CAnomalyJob::refreshMemoryAndReport() {
if (m_LastFinalisedBucketEndTime < m_ModelConfig.bucketLength()) {
LOG_ERROR(<< "Cannot report memory usage because last finalized bucket end time ("
<< m_LastFinalisedBucketEndTime << ") is smaller than bucket span ("
<< m_ModelConfig.bucketLength() << ')');
return;
}
// Make sure model size stats are up to date and then send a final memory
// usage report
for (const auto& detector_ : m_Detectors) {
Expand Down

0 comments on commit ad0b97e

Please sign in to comment.