checker memory leak

[ihard]

BUG REPORT

What version of Moira are you using ([binary] --version)?

checker -version
2023/09/23 17:26:12 maxprocs: Leaving GOMAXPROCS=32: CPU quota undefined
Moira Checker
Version: 2.8.4
Git Commit: 2f5f4b314998a7f0a6d00b689ca9b8b251bfbb0f

Configuration

checker.yaml

---

redis:
  addrs: "10.10.88.101:6380,10.10.88.101:6381"
  dbid: 0
  connection_limit: 2048
  metrics_ttl: 6h
  dial_timeout: 1s
  read_timeout: 10s
  write_timeout: 10s
telemetry:
  listen: ":8092"
  pprof:
    enabled: true
  graphite:
    enabled: true
    runtime_stats: false
    uri: "127.0.0.1:2003"
    prefix: metric-server-1.moira.info
    interval: 60s
checker:
  nodata_check_interval: 120s
  check_interval: 60s
  lazy_triggers_check_interval: 10m
  stop_checking_interval: 180s
  max_parallel_checks: 16
  max_parallel_remote_checks: 0
remote:
  enabled: false
log:
  log_file: stdout
  log_level: info

pprof heap + goroutine
pprof.zip

Metric trigger queue

Grafana Dashboard

What did you expect to see?

no memory leak

What did you see instead?

usage memory peak to 60GB and OOM

[kissken]

@ihard Привет, подскажи, пожалуйста, график с очередью - это в момент проблемы?

сможешь еще дополнить, пожалуйста, графиками?

aliasByNode(keepLastValue(nonNegativeDerivative(*.*.moira.*.checker.metricEventsHandle.count_ps), 1), 2)

keepLastValue(movingAverage(nonNegativeDerivative(*.*.moira.*.checker.loval.triggers.count_ps), '5min'), 1)

aliasByNode(*.*.moira.*.checker.metricEventsHandle.95-percentile, 2)

aliasByNode(*.*.moira.*.checker.local.triggers.95-percentile, 2)

[ihard]

@kissken added to first post

[ihard]

In pprof heap - problem function:

14676.30MB 96.79% 96.79% 14676.30MB 96.79% github.com/moira-alert/moira/metric_source.MakeEmptyMetricData (inline)

[kissken]

@ihard Could you, please, added info are triggers tagged or flat?
and yours triggers have system metrics without aggregate by pods?

I'm just guessing, in this case when many pods down and many up in one trigger, the problem appears or nor?

[ihard]

99% are flat triggers
tagged triggers all have fewer than 200 metrics
yes, there are some triggers with system metrics whose names are almost always constant
The problem is not related to the mass start or stop of the pods and appears continuously for 24 hours
It’s more likely that some trigger or triggers get into the checker, this causes a large memory consumption, then the process crashes and the trigger or triggers go to another node

[ihard]

Memory consumption when checking each trigger in debug mode or in metrics would greatly help in analyzing such problems.

[kissken]

hello, could you tell us, please, how many metrics match for pattern, when open /patterns page and sort by desc value at field metrics?

[ihard]

~ 100 000 at the time of analyzing the problems, now the number has grown to ~ 250,000
Also, right now the patterns page is not displayed, there is an error in the interface:
Load failed
in api log:
{"level":"info","module":"api","context":"http","http.method":"GET","http.uri":"http://127.0.0.1:7092/api/pattern","http.protocol":"HTTP/1.0","http.remote_addr":"10.225.88.101:50190","username":"anonymous","http.status":200,"http.content_length":377215140,"elapsed_time_ms":3478,"elapsed_time":"3.478224001s","time":"2023-10-09 22:46:16.994","message":"GET http://127.0.0.1:7092/api/pattern/ HTTP/1.0"}

[ihard]

Updating to 2.9.0 revealed about 100 division-by-0 triggers that were removed.
When updating to 2.9.0, a command was executed that removed all metrics from Redis:

cli -remove-all-metrics
cli -cleanup-last-checks
cli -cleanup-metrics
cli -cleanup-retentions
cli -cleanup-tags

Triggers with warning errors:
target t2 declared as alone metrics target but do not have any metrics and saved state in last check
have also been removed.
Now the checker logs are clean, but the problem persists.

Page patterns:

[ihard]

We removed a large pack of triggers for which no metrics were received, the problem stopped.

[almostinf]

In addition, I'd like to add that we found a potential issue that may be causing increased memory consumption - the moira-pattern-metrics key is not being cleared, which leads to a bunch of unnecessary requests in redis

Temporary fix for the problem:

1. SET DEL in triggers automatically clears this key for the trigger
2. Manually deleting metrics or delete all NODATA metrics also clears the key for the trigger

[ihard]

It's very similar to our problem, after deleting the metrics, everything goes back to normal for a while, but then the problem returns.

[almostinf]

Hi! We added the moira-pattern-metrics key cleanup to the cli command --cleanup-pattern-metrics, which will greatly reduce the load on the checker and the number of resources consumed, since the moira-pattern-metrics key leak generated unnecessary database queries

[ihard]

Will the Cli clean the keys in the current database, and the leak itself was previously fixed in some release?

[almostinf]

Yes, the command in the cli will clean up the current garbage, the leak fix will come in the next release. We recommend adding --cleanup-pattern-metrics to cli in the regular cronjob