File Rotation

File Rotation & Retention

This page explains how logme keeps file logs manageable in long-running applications.

There are two related but separate mechanisms:

• per-backend lifecycle policy in FileBackend;
• home directory size monitoring through DirectorySizeWatchdog.

They solve different problems and can be used together.

---

FileBackend lifecycle policy

FileBackend writes to one active file and can complete that file into an archive when a configured lifecycle boundary is reached. A completed file may then be retained, compressed, or deleted by retention rules.

The normal write path remains focused on appending data. Archive-name scanning, retention cleanup, and compression submission happen on startup cleanup or file-completion paths, not on every log record.

A production-style file backend can look like this:

{
  "type": "FileBackend",
  "file": "logs/app.log",
  "append": true,
  "rotation": "daily",
  "max-size": "100Mb",
  "on-size-limit": "rotate",
  "archive": "logs/archive/app.{date}.{index}.log",
  "compression": "gz",
  "retention": {
    "max-files": 14,
    "max-age": "30d",
    "max-total-size": "2Gb",
    "clean-on-start": true
  }
}

If a configured path is relative, it is resolved relative to the logger home directory.

See also File Backend and Configuration JSON.

---

Size control

max-size limits the active file size. By default, logme keeps the historical behavior: when the limit is exceeded, the active file is truncated rather than archived.

{
  "type": "FileBackend",
  "file": "logs/app.log",
  "max-size": "100Mb",
  "on-size-limit": "truncate"
}

max-size accepts either a plain byte count or byte-size suffixes documented in Utility Helpers, for example 512Kb, 64Mb, or 1Gb.

---

Size-based archive rotation

To rotate instead of truncating, set on-size-limit to rotate and provide an archive pattern containing {index}:

{
  "type": "FileBackend",
  "file": "logs/app.log",
  "max-size": "100Mb",
  "on-size-limit": "rotate",
  "archive": "logs/archive/app.{index}.log"
}

When the size limit is reached, the active file is closed, moved to the next free archive name, and the active file is opened again. Existing .log and .log.gz archive names are skipped, so runtime collisions and restart recovery do not overwrite older archives.

on-size-limit: "rotate" requires an archive pattern with {index}.

---

Time rotation

The rotation option accepts:

• none, off, or disabled;
• hourly;
• daily;
• weekly;
• monthly.

Example:

{
  "type": "FileBackend",
  "file": "logs/app.log",
  "rotation": "daily",
  "archive": "logs/archive/app.{date}.{index}.log"
}

The old rotation: "daily" behavior remains supported. If time rotation is disabled, the hot path does not compute time-period boundaries.

---

Combined time and size rotation

Size and time rotation can be used together. Both use the same file-completion path, so archive naming, retention, compression, and counters are handled consistently.

{
  "type": "FileBackend",
  "file": "logs/app.log",
  "rotation": "daily",
  "max-size": "100Mb",
  "on-size-limit": "rotate",
  "archive": "logs/archive/app.{date}.{index}.log"
}

Archive patterns can use:

• {index} — monotonically increasing archive part number for the current pattern/period;
• {date} — date part based on the completion period;
• {datetime} — date-time part based on the completion period.

When a time placeholder and {index} are used together, startup recovery restores the index for the current period. Archives from other periods do not consume the current period index.

---

Retention

Retention rules apply to completed archive files, not to the active file.

{
  "type": "FileBackend",
  "file": "logs/app.log",
  "rotation": "daily",
  "archive": "logs/archive/app.{date}.{index}.log",
  "retention": {
    "max-files": 14,
    "max-age": "30d",
    "max-total-size": "2Gb",
    "clean-on-start": true
  }
}

Retention fields:

• retention.max-files keeps only the newest matching archive files;
• retention.max-age removes matching archives older than the configured interval;
• retention.max-total-size keeps matching archive files under the configured total size by deleting older files first;
• retention.clean-on-start runs retention when the backend is configured.

Disabled values:

• max-files = 0 keeps all matching archives;
• max-age = 0 disables age-based cleanup;
• max-total-size = 0 disables total-size cleanup.

max-parts remains supported as a legacy alias for retention.max-files. If both are specified, they must have the same value.

Retention ignores unrelated files and protects the active file even if its name matches the archive cleanup pattern.

---

Compression

Optional gzip compression is enabled with compression: "gz" or compression: "gzip":

{
  "type": "FileBackend",
  "file": "logs/app.log",
  "rotation": "daily",
  "archive": "logs/archive/app.{date}.{index}.log",
  "compression": "gz"
}

Compression is submitted only for completed archive files. The active file is not compressed.

Gzip support depends on the USE_ZLIB CMake option. If zlib support is not compiled in, compression: "gz" is accepted but has no effect.

Accepted disabled values are none, off, disabled, or an empty string.

---

Lifecycle counters

When FILE_ENABLE_COUNTERS is enabled, FileBackend::GetCounters() and the [FileBackend] statistics dump include lifecycle counters:

• SizeLimitCompletionCalls — file completions triggered by size-limit rotation;
• TimeLimitCompletionCalls — file completions triggered by time rotation;
• ArchivedFiles — completed files successfully moved to archive names;
• CompressionSubmitCalls — completed archive files submitted to the compression manager;
• RetentionRuns — retention cleanup passes started by startup cleanup or file completion.

These counters are updated only on lifecycle paths. Normal appends do not run retention, compression, archive scanning, or filesystem cleanup.

---

Failure behavior

File rotation is best-effort. If an archive directory cannot be created or the active file cannot be moved to an archive name, the backend reopens the active file in append mode and keeps logging when possible.

This avoids losing existing active-log content on rotation failure.

---

Directory size watchdog

DirectorySizeWatchdog works at the logger home-directory level.

It is configured through the home-directory section:

{
  "home-directory": {
    "path": "logs",
    "watch-dog": {
      "enable": true,
      "max-size": "10GB",
      "check-periodicity": "30s",
      "file-extension": [".log", ".gz"]
    }
  }
}

The watchdog monitors total directory size and removes old files when the configured limit is exceeded.

This is different from per-backend retention:

• retention.* applies to archives produced by one FileBackend archive pattern;
• the watchdog controls total disk usage in the logging directory.

The watchdog consults the file manager before deleting files, so active files used by registered FileBackend instances are not removed.

---

Choosing the right mechanism

Use time rotation when log files should naturally split by time period.

Use on-size-limit: "rotate" when large active files should be completed into archives instead of truncated.

Use retention.* when each backend should retain only a bounded archive history.

Use the directory watchdog when total disk usage matters more than the number of archives per backend.

In production, it is common to combine all of them.

---

Related pages

• File Backend
• Configuration JSON
• Threading Model
• Performance