Metrics reporting unusually high number of written records

[korthout]

Describe the bug

The Grafana dashboard is showing an unusually high number of written records (~3M compared to ~3K per minute). See screenshots.

This seems to coincide with the restart of broker zeebe-2, which at the time of the restart was leader for partition 2. Partitions 1 and 3 were lead by zeebe-0, which took over leadership of partition 2 after zeebe-2 went down.

Screenshots

Larger timeframe

Shows high peaks in processing per partition panel. Also note that written normally is about 3K per minute per partition.

Focussed timeframe

Shows unusually high numbers as written in the processing per partition panel.

To Reproduce

Unknown.

I've tried to reproduce it on the same benchmark by restarting pods individually, but without success.

Expected behavior

Failover should not lead to a report of additional written records

Environment:

• Context: gke_zeebe-io_europe-west1-b_zeebe-cluster
• Namespace: medic-cw-23-f01b5bc83-benchmark
• Zeebe Version: f01b5bc
• Configuration: benchmark

[korthout]

Also seen in medic-cw-22-ce77f891b-benchmark on 2021-06-10 at 15:33:00 CEST -> grafana

Note, there is a leader change happening at that time, to make it visible in the dashboard, edit the role changes panel, and set the min interval of the prometheus query options to 1s.

[korthout]

Looks like the lastWrittenEventPosition is low (e.g. 0 or -1), see: https://github.com/camunda-cloud/zeebe/blob/develop/engine/src/main/java/io/camunda/zeebe/engine/processing/streamprocessor/ProcessingStateMachine.java#L366-L367

[Zelldon]

The issue is that we calculate the amount here https://github.com/camunda-cloud/zeebe/blob/develop/engine/src/main/java/io/camunda/zeebe/engine/processing/streamprocessor/ProcessingStateMachine.java#L368

But the lastWrittenEventPosition can be zero on fail-over, which means that the amount is very high, since the writtenEventPosition is quite high.

Ideally we would set the lastWrittenEventPosition on fail-over, maybe to the last processed position (which we get from the state) or something similar.