Collapse

Collapse Logging

Collapse logging is a call-site feature for reducing repeated log noise without hiding the first occurrence of a problem.

It is implemented by the LogmeX_Collapse and LogmeX_CollapseIgnore macro families in Logme.h, CollapseContextCache in Context.h, and Context::ApplyCollapse() in Context.cpp.

Macro families

The printf-style macros are:

Level	Exact-message collapse	Regex-normalized collapse
Debug	LogmeD_Collapse(limit, ...)	LogmeD_CollapseIgnore(ignoreRegex, limit, ...)
Info	LogmeI_Collapse(limit, ...)	LogmeI_CollapseIgnore(ignoreRegex, limit, ...)
Warning	LogmeW_Collapse(limit, ...)	LogmeW_CollapseIgnore(ignoreRegex, limit, ...)
Error	LogmeE_Collapse(limit, ...)	LogmeE_CollapseIgnore(ignoreRegex, limit, ...)
Critical	LogmeC_Collapse(limit, ...)	LogmeC_CollapseIgnore(ignoreRegex, limit, ...)

X in the macro name is the normal log level letter: D, I, W, E, or C.

Collapse macros accept the same optional log arguments as normal LogmeX macros after the collapse-specific parameters: channel, subsystem, override, format string, and format arguments.

LogmeX_Collapse

LogmeX_Collapse(limit, ...) uses the final formatted message text as the repeat key.

LogmeW_Collapse(3, "network error: connection refused");

Behavior:

1. The first message from this macro call site is written normally.
2. Later calls from the same macro call site that produce the same comparison key are not written immediately.
3. When the repeat counter reaches limit, logme writes a collapsed summary using the original message text.
4. When the same macro call site produces a different comparison key, the stored key is replaced and the new message is written normally.

This is a call-site cache, not a backend-wide consecutive-line filter. Other log records written by other source lines, functions, threads, or channels do not reset this call site's collapse counter.

The summary text is produced by the logger formatting path and has the form:

repeated N times: original message text

What collapse does not do

Collapse does not detect or aggregate a repeated sequence of several different log records in the final log file. It works with one formatted message at one macro call site.

If a logical operation is logged as several separate records, there are two typical choices:

1. Put _Collapse / _CollapseIgnore on each repeated log statement. Each line will keep its own counter and will be summarized independently.
2. If the whole block really should be treated as one diagnostic record, build one formatted message that contains embedded \n characters and collapse that single message.

For example, four MQTT reconnect lines such as Connecting, Connected, Subscribe, and Disconnected will not become one "block repeated 25 times" automatically. They can be collapsed as four independent call sites, or the code can write one multi-line reconnect diagnostic if a block-level summary is what is desired.

LogmeX_CollapseIgnore

LogmeX_CollapseIgnore(ignoreRegex, limit, ...) is intended for repeated messages that contain volatile fields such as request ids, correlation ids, timestamps, sequence numbers, counters, or addresses.

Before comparing the current message with the previous one, logme removes all substrings matched by ignoreRegex from the formatted message text. The regular expression is used only to build the comparison key. The original formatted message is still what gets printed.

LogmeE_CollapseIgnore(
  "request_id=[0-9]+"
  , 10
  , "request_id=%llu backend unavailable"
  , requestId
);

This can collapse messages such as:

request_id=101 backend unavailable
request_id=102 backend unavailable
request_id=103 backend unavailable

because the comparison key becomes the same after removing the request id.

Important details

• Collapse state is stored in a static CollapseContextCache at the macro call site. Different source lines have independent collapse state.
• The comparison key is based on formatted text, not on the format string and raw arguments. The usual logger prefix, such as timestamp and thread id, is added after the collapse decision.
• Repeats are counted for the same macro call site and comparison key. Other log messages in the file do not reset the counter. A different comparison key from the same call site resets that call-site state.
• The limit value is repeat-count based, not time based. For example, with limit == 20, the first occurrence is printed immediately and the next summary is printed only after 20 matching repeat attempts.
• logme does not currently flush a final "pending repeats" summary when the log is closed. If the limit is not reached before shutdown, the suppressed repeat count is not written.
• limit == 0 disables collapse for that call site and the message is written normally.
• The collapse cache is protected by its own lock, so the repeat counter and last key are shared safely between threads using the same call site.
• LogmeX_CollapseIgnore compiles the regular expression in the static call-site cache. If the regular expression is invalid, ignore matching is disabled and the macro behaves like exact-message collapse.

When to use it

Use collapse when a message is important, but identical or nearly identical repeats would flood the log.

Good candidates:

• retry loops that report the same failure repeatedly
• temporary backend outages
• polling loops
• noisy network or protocol errors
• repeated validation failures where only ids or timestamps differ

Choose the limit according to how often you still want progress information. If a retry happens once per second and limit is 60, the summary is emitted roughly once per minute while the same condition continues. If every reconnect or every retry must be visible, do not collapse that message, or use a time-based _Every(ms, ...) macro instead.

Do not use collapse for messages where every individual occurrence must be preserved, such as audit records, security decisions, billing events, or one-record-per-request traces.

Difference from _Once and _Every

_Once, _Every, and _Collapse solve different noise problems:

Need	Recommended macro
Write only the first occurrence	LogmeX_Once(...)
Write at most once per time interval	LogmeX_Every(ms, ...)
Write the first occurrence and aggregate repeated attempts at the same call site	LogmeX_Collapse(limit, ...)
Aggregate repeats after ignoring volatile fields	LogmeX_CollapseIgnore(ignoreRegex, limit, ...)

Collapse is not a backend-side duplicate filter. It is an explicit call-site decision, so the repeated message can be suppressed before backend delivery.