Log rotation for k3s log

[agelwarg]

During testing, we recently discovered this bug in k3s:

k3s-io/k3s#1628

As a side effect, the k3s log at /var/log/k3s-service.log is growing large very quickly. This bug is one problem, but there could be others or just normal growth to fill the filesystem over time. On a system with constrained storage, this is problematic. To protect against this, it would be nice to include log rotation in some form. Maybe it makes sense just to use syslog?

[agelwarg]

In my testing, it doesn't look like supervise-daemon supports the output_logger or error_logger settings, whereas start-stop-daemon does. I also found this issue for OpenRC: OpenRC/openrc#341

[agelwarg]

I see there is already a logrotate config file, but crond is not started on boot. Is that by design or just a miss?

[dweomer]

I see there is already a logrotate config file, but crond is not started on boot. Is that by design or just a miss?

@agelwarg probably an intentional miss in that a boot_cmd or run_cmd can be specified to start the service (assuming it is configured correctly).

[philipsparrow]

My workaround solution to this is as follows:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: logrotate
  namespace: k3os-system
  labels:
    app: logrotate
spec:
  selector:
    matchLabels:
      app: logrotate
  template:
    metadata:
      labels:
        app: logrotate
    spec:
      automountServiceAccountToken: false
      initContainers:
      - name: truncate
        # Any very large log files should be truncated before rotation this saves disk
        # copytruncate (the default logrotate option) copies the file before truncating
        # e.g. 3GB k3s-service.log file becomes 6GB across files during the logrotate operation
        image: debian:10-slim # This is a scaled down image which contains coreutils 8.30
        command: ["truncate", "--size=<30M", "/host/var/log/k3s-service.log"]
        volumeMounts:
        - name: logs
          mountPath: /host/var/log
        securityContext:
          privileged: true
          runAsUser: 0
          capabilities:
            drop: ["ALL"]
      containers:
      - name: logrotate
        image: quay.io/honestbee/logrotate:latest
        env:
        - name: LOGROTATE_PATTERN
          value: /host/var/log/k3s-service.log
        - name: LOGROTATE_ROTATE
          value: "3" # Keep 3 files
        - name: LOGROTATE_SIZE
          value: 20M
        - name: CRON_SCHEDULE
          value: "0 5 * * *" # Every day at 05:00 UTC
        securityContext:
          # Requires write access to /etc in the pod and /var/log on the host
          privileged: true
          runAsUser: 0
          capabilities:
            drop: ["ALL"]
        resources:
          requests:
            cpu: 50m
            memory: 50Mi
          limits:
            cpu: 50m
            memory: 50Mi
        volumeMounts:
        - name: logs
          mountPath: /host/var/log
      volumes:
      - name: logs
        hostPath: # Mount the host directory
          type: Directory
          path: /var/log

[linuxmail]

hi @philipsparrow,

thanks for the workaround. My logfile was around 8GB and brought my test cluster down.

[sat0yu]

@agelwarg probably an intentional miss in that a boot_cmd or run_cmd can be specified to start the service (assuming it is configured correctly).

Confirmed that log rotation is able to be done as starting crond through run_cmd

run_cmd:
  - "service crond start"

Also, found that a default config file /etc/logrotate.d/k3s-server (for /var/log/k3s-server.log) is generated automatically every time when launching k3os.
That made it difficult to perform log rotation on /var/log/k3s-server.log. Because, even if another file already exists at /etc/logrotate.d/k3s-server, that default config file always overrides it.

So, one possible idea to use a custom configuration is placing a config file with a different name in /etc/logrotate.d.
Modified: sorry, this does not work because it causes an error error: k3s-service-log:1 duplicate log entry for /var/log/k3s-service.log

write_files:
  - content: |-
      /var/log/k3s-service.log {
              su root root
              maxsize 10M
              missingok
              notifempty
              copytruncate
      }
    owner: root
    path: /etc/logrotate.d/k3s-service-log
    permissions: "0644"

[sasanbabai]

I'm unable to change /etc/logrotate.d/k3s-server even using either boot_cmd or run_cmd:

run_cmd:
- sed -i "$ i size 10M" /etc/logrotate.d/k3s-server

I can confirm that the command runs as part of the boot but the file is overwritten after it is changed:

run_cmd:
- sed -i.back "$ i size 10M" /etc/logrotate.d/k3s-server

The above will create /etc/logrotate.d/k3s-server.back but changes to the original config is reverted. Surprisingly, this only happens with /etc/logrotate.d/k3s-server not any other config under /etc/logrotate.d nor for any file under the ephemeral /etc

[gravufo]

I just figured out that the logrotate config file for k3s-service is being created by the k3s install script. This overwrites anything we do in run_cmd, boot_cmd or write_files. We would need to do something that runs completely after everything (which I thought boot_cmd was, but seems I was wrong).

The daemonset approach is interesting, but not ideal since it requires to run something inside the cluster. If the cluster itself is broken and k3s is crashing in loop, the k3s log file will become huge and not get rotated because the logrotate config was not updated by the daemonset.

One way to fix this would be to change the default logrotate config, but that affects all log files so it's not ideal.

[dweomer]

I just figured out that the logrotate config file for k3s-service is being created by the k3s install script. This overwrites anything we do in run_cmd, boot_cmd or write_files. We would need to do something that runs completely after everything (which I thought boot_cmd was, but seems I was wrong).

That ... really sounds like an annoying deal, I'm sorry.

However, if the service-specific configs in the /etc/logrotate.d drop-in are read at job execution time, aka not at cron start-up, then we should be able to get by with a hack leveraging a script in the /etc/local.d drop-in location (for default configuration the local service is the last thing run at startup). See /etc/local.d/README on a running k3OS system:

This directory should contain programs or scripts which are to be run
when the local service is started or stopped.

If a file in this directory is executable and it has a .start extension,
it will be run when the local service is started. If a file is
executable and it has a .stop extension, it will be run when the local
service is stopped.

All files are processed in lexical order.

Keep in mind that files in this directory are processed sequentially,
and the local service is not considered started or stopped until
everything is processed, so if you have a process which takes a long
time to run, it can delay your boot or shutdown processing.

See also /etc/init.d/local

The daemonset approach is interesting, but not ideal since it requires to run something inside the cluster. If the cluster itself is broken and k3s is crashing in loop, the k3s log file will become huge and not get rotated because the logrotate config was not updated by the daemonset.

Agreed.

[gravufo]

Thanks for your response @dweomer. I can confirm that it works using a startup script inside /etc/init.d/local. Very appreciated :)

[philomory]

It would be great if we could come up with a setup for k3s-service that didn't require logrotate to run in copytruncate mode; right now it's not really feasible to both enable log rotation, and consume those logs with a tool like filebeat or fluent-bit. The only options right now seem to be:

1. Don't enable log rotation (bleh)
2. Enable log rotation with copytruncate enabled and accept that most log scrapers will fail (at least Filebeat and Fluent-bit)
3. Enable log rotation without copytruncate, and use a postrotate hook to restart the k3s service, which hardly seems ideal.

The funny thing is, the k3s binary has built-in log rotation already, and seems to have had it for quite a while; if you log to a file with --log /var/log/k3s-service.log or similar, it'll automatically rotate the log every 50 megabytes and/or 28 days, keeping the most recent 3. Now, this behavior isn't configurable, which is a bit of a shame, but it's there, and it native to the logging library used, which means you don't need to resort to copytruncate or sighup hacks.

Unfortunately, when the k3s install.sh script is run on an openrc-based distribution, it's configures k3s to be executed via the command:

supervisor=supervise-daemon
name=${SYSTEM_NAME}
command="${BIN_DIR}/k3s"
command_args="$(escape_dq "${CMD_K3S_EXEC}")
    >>${LOG_FILE} 2>&1"

output_log=${LOG_FILE}
error_log=${LOG_FILE}

This leaves k3s logging to stdout/stderr by default, and then redirects that output to a file (at multiple levels) rather than allowing the k3s logging code to do its job.

That said, I suppose if you use k3os.k3s_args within cloud config, you can set the --log argument there.