sentinal: Software for Logfile and Inode Management

System and application processes can create many files and large files, possibly causing disk partitions to run out of space. sentinal is a systemd service for managing files and filesystems to comply with the directives in an INI configuration file. Depending on the goals, sentinal can also act as an adjunct or an alternative to logrotate.

Monitoring and management capabilities:

• available filesystem disk space by percentage
• available filesystem inode usage by percentage or count
• expire files by size, age, or retention settings
• inodes by age or retention settings
• log ingestion, processing, and rotation
• monitor and process log files when they reach a given size

Configuration

sentinal uses INI files for its runtime configuration. Each section in the INI file pertains to one resource: a directory, possibly a file template, and the conditions for managing the resource.

INI File Description

An INI file must contain a section called global. This section must include a pidfile definition and an optional sqlite3 database definition. The database name can be :memory:, or a pathname of a disk file.

Section names can be up to 11 characters in length (kernel max), and the characters must be alpha-numeric or underscore (valid sqlite3 table name). Section names must be unique in the INI file.

[global]
pidfile:   sentinal process id and lock file, for manual logrotate
database:  name of the sqlite3 database

[section]
command:   command to run
dirname:   thread and postcmd working directory, file location
dirlimit:  maximum total size of matching files in a directory
subdirs:   option to search subdirectories for matching files (false)
pipename:  named pipe/fifo fifo location
template:  output file name, date(1) sequences %F %Y %m %d %H %M %S %s
pcrestr:   perl-compatible regex naming files to manage
uid:       username or uid for command/postcmd; default = nobody
gid:       groupname or gid for command/postcmd; default = nogroup
rotatesiz: rotate size, M = MiB, G = GiB; 0 = no rotate (unlimited)
expiresiz: expire size, M = MiB, G = GiB; 0 = no expiration by size
diskfree:  percent blocks free; 0 = no monitor (off)
inofree:   percent inodes free; 0 = no monitor (off)
expire:    file retention time, units = m, H, D, W, M, Y; 0 = no expiration (off)
retmin:    minimum number of files to retain; 0 = none (off)
retmax:    maximum number of files to retain; 0 = no max (off)
terse:     option to record or suppress file removal notices (false)
rmdir:     option to remove empty directories (false)
symlinks:  option to follow symlinks (false)
postcmd:   command to run after log closes or rotates, %file = filename
truncate:  option to truncate slm-managed files (false)

Thread Requirements

Threads need several keys defined in order for them to run. Below are the thread types and their required keys:

Diskfree (DFS) Thread

• pcrestr
• one or more of the following
  • diskfree
  • inofree
• optional
  • retmin

Expire (EXP) Thread

• pcrestr
• one or more of the following
  • dirlimit
  • expire
  • retmax
• optional
  • retmin

Simple log Monitor (SLM) Thread

• command unset (null)
• template
• postcmd
• rotatesiz
• optional, likely required by use case
  • uid
  • gid

Work (log ingestion, WRK) Thread

• command
• pipename
• template
• optional, but recommended
  • rotatesiz
• optional
  • postcmd
• optional, likely required by use case
  • uid
  • gid

Note the following conditions. If:

• command is set, template must be set.
• rotatesiz is set, rotate the file after it reaches the specified size.
• expiresiz is set, remove files larger than the specified size at the expiration time.
• diskfree is set, create a thread to discard the oldest files to free disk space.
• inofree is set, create a thread to discard the oldest files to free inodes.
• expire is set, remove files older than the expiration time.
• retmin is set, retain n number of files, regardless of expiration or available disk space.
• retmax is set, retain a maximum number of n files, regardless of expiration.
• postcmd is specified, the value is passed as a command to bash -c after the file closes or rotates. Optional.

Precedence of Keys:

• retmin, retmax take precedence over dirlimit, diskfree, inofree, expire.
• dirlimit, diskfree, inofree take precedence over expire.

Free Disk Space

sentinal can remove files when the filesystem they occupy falls below the free space constraint.

flowchart TB
    s1[ read diskfree ]
    s2[ check diskfree ]
    d1{ low free space }
    a1[ yes ]
    a2[ no ]
    s3[ remove oldest files ]
    s9[ return to check diskfree ]
    s1 --> s2 --> d1
    d1 --> a1 --> s3 --> s9
    d1 --> a2 --> s9

Example: to monitor console logs in /opt/sentinal/log for 20% free disk space:

[global]
pidfile   = /run/diskfree.pid
database  = :memory:

[console]
dirname   = /opt/sentinal/log
pcrestr   = console
diskfree  = 20

File Expiration

sentinal can remove files when they meet one or more of the following constraints: retmin, retmax, expire, expiresiz, dirlimit.

The combinations of expire and expiresiz settings affect expiration behavior. If:

• expire is set and expiresiz is unset, remove files older than the expiration time.
• expire and expiresiz are set, remove files larger than expiresiz at the expiration time.
• expiresiz is set and expire is unset, take no action.

flowchart TB
    s1[ read<br>retmin, retmax<br>expire, expiresiz<br>dirlimit ]
    s2[ check vars ]
    d1{ min retention }
    d2{ max retention }
    d3{ dir size limit }
    d4{ expiration time<br>or size }
    a1[ yes ]
    a2[ no ]
    a3[ yes ]
    a4[ no ]
    a5[ yes ]
    a6[ no ]
    a7[ yes ]
    a8[ no ]
    s3[ remove oldest or expired files ]
    s9[ return to check vars ]
    s1 --> s2
    s2 --> d1 --> a1 --> s9
    d1 --> a2 --> d2 --> a3 --> s3 --> s9
    d2 --> a4 --> d3 --> a5 --> s3
    d3 --> a6 --> d4 --> a7 --> s3
    d4 --> a8 --> s9

Expiration example: This INI configuration removes gzipped files in /var/log and its subdirectories after two weeks:

[global]
pidfile   = /run/varlog.pid
database  = :memory:

[zipped]
dirname   = /var/log
subdirs   = true
pcrestr   = \.gz
expire    = 2w

Expiration example: This INI uses two threads to remove compressed files in /sandbox. sandbox2M removes compressed files aged two months or older. sandbox1M removes compressed files aged one month or older if their sizes exceed 10GiB, logging the removals.

[global]
pidfile   = /run/sandbox.pid
database  = :memory:

[sandbox2M]
dirname   = /sandbox
subdirs   = true
pcrestr   = \.(bz2|gz|lz|zip|zst)
expire    = 2M

[sandbox1M]
dirname   = /sandbox
subdirs   = true
pcrestr   = \.(bz2|gz|lz|zip|zst)
rotatesiz = 10G
expiresiz = 10G
expire    = 1M
terse     = false

Directory usage example: Remove myapp logs matching myapplog-\d{8}$ when they consume more than 500MiB of disk space or the number of logs exceeds 21:

[global]
pidfile   = /run/myapplog.pid
database  = :memory:

[myapp]
dirname   = /var/log/myapp
dirlimit  = 500M
pcrestr   = myapplog-\d{8}$
retmax    = 21

Simple Log Monitor

sentinal, using inotify, can monitor and process logs when they reach a specified size. A sentinal section for SLM must not set command. The keys template, postcmd, and rotatesiz must be set.

flowchart TB
    s1[ read rotatesiz, postcmd ]
    s2[ check size ]
    d1{ size reached }
    a1[ yes ]
    a2[ no ]
    s3[ run postcmd ]
    s4[ return to check size ]
    s1 --> s2
    s2 --> d1
    d1 --> a1
    d1 --> a2
    a1 --> s3
    s3 --> s4
    a2 --> s4

In this example, sentinal runs logrotate on chattyapp.log when the log exceeds 50MiB in size:

[global]
pidfile   = /run/chattyapp.pid
database  = :memory:

[chattyapp]
dirname   = /var/log
template  = chattyapp.log
uid       = root
gid       = root
rotatesiz = 50M
postcmd   = /usr/sbin/logrotate -f /opt/sentinal/etc/chattyapp.conf

This example is the same as above, adding a 20% diskfree check for logs processed by logrotate:

[global]
pidfile   = /run/chattyapp.pid
database  = :memory:

[chattyapp]
dirname   = /var/log
template  = chattyapp.log
pcrestr   = chattyapp\.log\.\d
uid       = root
gid       = root
rotatesiz = 50M
diskfree  = 20
postcmd   = /usr/sbin/logrotate -f /opt/sentinal/etc/chattyapp.conf

Logfile Ingestion and Processing

sentinal can ingest and process logs, rotate them on demand or when they reach a specified size, and optionally post-process logs after rotation. For logfile processing, replace the application's logfile with a FIFO, and set sentinal to read from it.

sequenceDiagram
    participant Application
    participant FIFO
    participant Sentinal
    participant Logfile
    Application ->> FIFO: Application writes to FIFO
    FIFO ->> Sentinal: Sentinal reads from FIFO
    Sentinal ->> Logfile: Sentinal creates logfile
    Sentinal ->> Sentinal: Sentinal auto-rotates logfile
    Sentinal -> Logfile: Optionally post-process logfile

For example, this configuration connects the dd program to example.log for log ingestion, and rotates and compresses the log when it reaches 5GiB in size:

[example]
command   = /bin/dd bs=64K status=none
dirname   = /var/log
pipename  = example.log
template  = example-%Y-%m-%d_%H-%M-%S.log
pcrestr   = example-
uid       = appowner
gid       = appgroup
rotatesiz = 5G
postcmd   = /usr/bin/zstd --rm %file 2>/dev/null

This example does basically the same as above, but with inline compression (no intermediate files), and rotates the compressed log when it reaches 1GiB in size:

[example]
command   = /usr/bin/zstd
dirname   = /var/log
pipename  = example.log
template  = example-%Y-%m-%d_%H-%M-%S.log.zst
pcrestr   = example-
uid       = appowner
gid       = appgroup
rotatesiz = 1G

systemd unit file

sentinal runs as a systemd service. The following is an example of a unit file:

[Unit]
Description=Shim to zstd-compress logs
StartLimitIntervalSec=0
StartLimitBurst=10
After=network.target network-online.target systemd-networkd.service

[Service]
Type=simple
Restart=always
RestartSec=2
User=root
ExecStart=/opt/sentinal/bin/sentinal -f /opt/sentinal/etc/sentinal.ini
ExecReload=/bin/kill -s HUP $MAINPID

[Install]
WantedBy=multi-user.target

User/Group ID Notes

• User/Group ID applies only to command and postcmd; otherwise, sentinal runs as the calling user.

• If an application never needs root privileges to run and process logs, consider setting and using the application's user and group IDs in the unit file.

• Running sentinal as root is likely necessary when a single sentinal instance monitors several different applications.

• When unspecified, the user and group IDs are set to nobody and nogroup.

Exported Environment Variables

sentinal exports the following variables to command and postcmd:

HOME       home of uid, default /tmp
PATH       /usr/bin:/usr/sbin:/bin
SHELL      /bin/bash
PWD        dirname value from INI file
TEMPLATE   template value from INI file
PCRESTR    pcrestr value from INI file

Sentinal Status

The INI file /opt/sentinal/etc/example.ini is used here as an example.

# systemctl status sentinal
* sentinal.service - sentinal service for example.ini
     Loaded: loaded (/etc/systemd/system/sentinal.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-11-24 13:01:47 PST; 4s ago
   Main PID: 13580 (sentinal)
      Tasks: 4 (limit: 76930)
     Memory: 852.0K
     CGroup: /system.slice/sentinal.service
             `-13580 /opt/sentinal/bin/sentinal -f /opt/sentinal/etc/example.ini

Nov 24 13:01:47 loghost sentinal[13580]: test4: start dfs thread: /opt/sentinal/tests
Nov 24 13:01:47 loghost sentinal[13580]: test4: monitor disk: / for 85.00% free
Nov 24 13:01:47 loghost sentinal[13580]: test4: monitor file: test4- for retmin 3
Nov 24 13:01:47 loghost sentinal[13580]: test4: /opt/sentinal/tests: 87.06% blocks free
Nov 24 13:01:47 loghost sentinal[13580]: test4: start exp thread: /opt/sentinal/tests
Nov 24 13:01:47 loghost sentinal[13580]: test4: monitor file: test4- for retmin 3
Nov 24 13:01:47 loghost sentinal[13580]: test4: monitor file: test4- for retmax 25
Nov 24 13:01:47 loghost sentinal[13580]: test4: start wrk thread: /opt/sentinal/tests
Nov 24 13:01:47 loghost sentinal[13580]: test4: command: /usr/bin/zstd -T4
Nov 24 13:01:47 loghost sentinal[13580]: test4: monitor file: test4- for size 1024MiB

(In this example, /opt is in the / filesystem)

Build, Install

sentinal requires the pcre2-devel package for building the software.

# cd sentinal
# make
# make install

Create a systemd unit file and add it to the local systemd directory, or run

# make systemd

to install an example as a starting point.

Edit /etc/systemd/system/sentinal.service as necessary.

# systemctl daemon-reload

Test INI Files

sentinal provides two options for testing INI files. -d|--debug prints INI file sections as parsed, where the output is similar to the input. -v|--verbose prints INI file sections with the keys evaluated as they would be at run time, including symlink resolution and relative to full pathname conversion.

Run

# systemctl enable sentinal
# systemctl start sentinal

Useful commands for monitoring sentinal:

# journalctl -f -n 20 -t sentinal
# journalctl -f _SYSTEMD_UNIT=example.service
$ ps -lT -p $(pidof sentinal)
$ top -H -S -p $(echo $(pgrep sentinal) | sed 's/ /,/g')
$ htop -d 5 -p $(echo $(pgrep sentinal) | sed 's/ /,/g')
# lslocks -p $(pidof sentinal)
# pmap -X $(pidof sentinal)

Examples of on-demand log rotation:

# systemctl reload sentinal
# pkill -HUP sentinal
# kill -HUP $(cat /path/to/pidfile)

Notes

• Linux processes writing to pipes block when processes are not reading from them. systemd manages sentinal to ensure sentinal is always running. See README.fifo for additional information about FIFO behavior.

• The default pipe size in Linux is either 64KB or 1MB. sentinal increases its pipe sizes on 3.x.x and newer kernels to 64MiB. Consider this a tuning parameter that can affect performance.

• In the inline compression example, zstd can be changed to a different program, e.g., gzip or (p)bzip2, though they are slower and may impact the performance of the writer application.

• For inode management, sentinal counts inodes in dirname, not inodes in the filesystem.

• sentinal reports free space for unprivileged users, which may be less than privileged users' values reported by disk utility programs.

• The rotatesiz key represents bytes written to disk. When command is a compression program, log rotation occurs after sentinal writes rotatesiz bytes post-compression. If rotatesiz is unset or zero, the thread requires manual or cron-based log rotation.

• sentinal removes empty directories within dirname when rmdir is true. To preserve a single directory, create a file in the directory with a file name that does not match pcrestr, for example, .persist.

• sentinal does not descend into directories in other filesystems, similar to find dir -xdev.