Collapse

Collapse Logging

Collapse logging is a call-site feature for reducing repeated log noise without hiding the first occurrence of a problem.

It is implemented by the LogmeX_Collapse, LogmeX_CollapseEvery, LogmeX_CollapseIgnore, and LogmeX_CollapseIgnoreEvery macro families in Logme.h, CollapseContextCache in Context.h, and Context::ApplyCollapse() in Context.cpp.

Macro families

The printf-style macros are:

Level	Count-based collapse	Time-based collapse	Count-based regex-normalized collapse	Time-based regex-normalized collapse
Debug	LogmeD_Collapse(limit, ...)	LogmeD_CollapseEvery(intervalMs, ...)	LogmeD_CollapseIgnore(ignoreRegex, limit, ...)	LogmeD_CollapseIgnoreEvery(ignoreRegex, intervalMs, ...)
Info	LogmeI_Collapse(limit, ...)	LogmeI_CollapseEvery(intervalMs, ...)	LogmeI_CollapseIgnore(ignoreRegex, limit, ...)	LogmeI_CollapseIgnoreEvery(ignoreRegex, intervalMs, ...)
Warning	LogmeW_Collapse(limit, ...)	LogmeW_CollapseEvery(intervalMs, ...)	LogmeW_CollapseIgnore(ignoreRegex, limit, ...)	LogmeW_CollapseIgnoreEvery(ignoreRegex, intervalMs, ...)
Error	LogmeE_Collapse(limit, ...)	LogmeE_CollapseEvery(intervalMs, ...)	LogmeE_CollapseIgnore(ignoreRegex, limit, ...)	LogmeE_CollapseIgnoreEvery(ignoreRegex, intervalMs, ...)
Critical	LogmeC_Collapse(limit, ...)	LogmeC_CollapseEvery(intervalMs, ...)	LogmeC_CollapseIgnore(ignoreRegex, limit, ...)	LogmeC_CollapseIgnoreEvery(ignoreRegex, intervalMs, ...)

X in the macro name is the normal log level letter: D, I, W, E, or C.

Collapse macros accept the same optional log arguments as normal LogmeX macros after the collapse-specific parameters: channel, subsystem, override, format string, and format arguments.

LogmeX_Collapse

LogmeX_Collapse(limit, ...) uses the final formatted message text as the repeat key.

LogmeW_Collapse(3, "network error: connection refused");

Behavior:

1. The first message from this macro call site is written normally.
2. Later calls from the same macro call site that produce the same comparison key are not written immediately.
3. When the repeat counter reaches limit, logme writes a collapsed summary using the original message text.
4. When the same macro call site produces a different comparison key, the stored key is replaced and the new message is written normally.

This is a call-site cache, not a backend-wide consecutive-line filter. Other log records written by other source lines, functions, threads, or channels do not reset this call site's collapse counter.

The summary text is produced by the logger formatting path and has the form:

repeated N times: original message text

LogmeX_CollapseEvery

LogmeX_CollapseEvery(intervalMs, ...) uses the same comparison key as LogmeX_Collapse, but the repeat summary is controlled by time instead of a repeat counter.

LogmeW_CollapseEvery(1000, "backend connection failed; retrying");

The first message is written normally. Matching repeated messages from the same macro call site are suppressed while the interval is still active. When intervalMs has elapsed and the same message is emitted again, logme writes one message with the accumulated repeat count and starts a new interval.

The output format is the same as count-based collapse:

repeated N times: original message text

This is useful when a message can be emitted in a very tight loop. A count-based limit such as 1000 can still produce too many records if the loop is extremely fast. A time-based collapse such as CollapseEvery(1000, ...) limits the visible output to roughly one collapsed record per second for that macro call site and comparison key.

If the comparison key changes while repeats are pending, logme does not print a pending summary for the old key. The new key simply starts a new collapse series and is written normally.

intervalMs == 0 disables time-based collapse for that call site and the message is written normally.

What collapse does not do

Collapse does not detect or aggregate a repeated sequence of several different log records in the final log file. It works with one formatted message at one macro call site.

If a logical operation is logged as several separate records, there are two typical choices:

1. Put _Collapse, _CollapseEvery, _CollapseIgnore, or _CollapseIgnoreEvery on each repeated log statement. Each line will keep its own state and will be summarized independently.
2. If the whole block really should be treated as one diagnostic record, build one formatted message that contains embedded \n characters and collapse that single message.

For example, four MQTT reconnect lines such as Connecting, Connected, Subscribe, and Disconnected will not become one "block repeated 25 times" automatically. They can be collapsed as four independent call sites, or the code can write one multi-line reconnect diagnostic if a block-level summary is what is desired.

LogmeX_CollapseIgnore and LogmeX_CollapseIgnoreEvery

LogmeX_CollapseIgnore(ignoreRegex, limit, ...) and LogmeX_CollapseIgnoreEvery(ignoreRegex, intervalMs, ...) are intended for repeated messages that contain volatile fields such as request ids, correlation ids, timestamps, sequence numbers, counters, or addresses.

Before comparing the current message with the previous one, logme removes all substrings matched by ignoreRegex from the formatted message text. The regular expression is used only to build the comparison key. The original formatted message is still what gets printed.

LogmeE_CollapseIgnoreEvery(
  "request_id=[0-9]+"
  , 1000
  , "request_id=%llu backend unavailable"
  , requestId
);

This can collapse messages such as:

request_id=101 backend unavailable
request_id=102 backend unavailable
request_id=103 backend unavailable

because the comparison key becomes the same after removing the request id.

Important details

• Collapse state is stored in a static CollapseContextCache at the macro call site. Different source lines have independent collapse state.
• The comparison key is based on formatted text, not on the format string and raw arguments. The usual logger prefix, such as timestamp and thread id, is added after the collapse decision.
• Repeats are counted for the same macro call site and comparison key. Other log messages in the file do not reset the counter.
• Count-based collapse writes a summary when the repeat counter reaches limit.
• Time-based collapse writes a summary when the same comparison key is emitted again after intervalMs has elapsed.
• If the comparison key changes, pending suppressed repeats for the old key are discarded and no extra summary record is generated.
• logme does not currently flush a final "pending repeats" summary when the log is closed. If the count limit is not reached or the interval does not expire before shutdown, the suppressed repeat count is not written.
• limit == 0 disables count-based collapse for that call site and the message is written normally.
• intervalMs == 0 disables time-based collapse for that call site and the message is written normally.
• The collapse cache is protected by its own lock, so the repeat counter, last key, and last output time are shared safely between threads using the same call site.
• Ignore variants compile the regular expression in the static call-site cache. If the regular expression is invalid, ignore matching is disabled and the macro behaves like exact-message collapse for the same count-based or time-based mode.

When to use it

Use collapse when a message is important, but identical or nearly identical repeats would flood the log.

Good candidates:

• retry loops that report the same failure repeatedly
• temporary backend outages
• polling loops
• noisy network or protocol errors
• repeated validation failures where only ids or timestamps differ

Choose count-based collapse when you want a summary after a known number of repeated attempts. Choose time-based collapse when the message can be emitted very frequently and the log should not contain more than one collapsed record per time window.

If a retry happens once per second and limit is 60, LogmeW_Collapse(60, ...) emits a summary roughly once per minute. If the same message can be emitted thousands of times per second, LogmeW_CollapseEvery(1000, ...) is usually a better fit.

Do not use collapse for messages where every individual occurrence must be preserved, such as audit records, security decisions, billing events, or one-record-per-request traces.

Difference from _Once and _Every

_Once, _Every, and _Collapse solve different noise problems:

Need	Recommended macro
Write only the first occurrence	LogmeX_Once(...)
Write at most once per time interval without a repeat summary	LogmeX_Every(ms, ...)
Write the first occurrence and aggregate repeated attempts by count	LogmeX_Collapse(limit, ...)
Write the first occurrence and aggregate repeated attempts by time	LogmeX_CollapseEvery(intervalMs, ...)
Aggregate repeats after ignoring volatile fields	LogmeX_CollapseIgnore(...) / LogmeX_CollapseIgnoreEvery(...)

Collapse is not a backend-side duplicate filter. It is an explicit call-site decision, so the repeated message can be suppressed before backend delivery.