Skip to content

Error when using journald log driver and FallbackToLogsOnError #52502

@joelsmith

Description

@joelsmith

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

When using the legacy docker container runtime and when a container has terminationMessagePolicy=FallbackToLogsOnError and when docker is configured with a log driver other than json-log (such as journald), the kublet still tries to fetch the last few lines of the container logs from the json log file. When it tries, it gets a message like this:

Error on reading termination message from logs: failed to open log file "/var/log/pods/f36cd8db-9984-11e7-9d0f-54e1ad486deb/busybox_0.log": open /var/log/pods/f36cd8db-9984-11e7-9d0f-54e1ad486deb/busybox_0.log: no such file or directory

What you expected to happen:

The kubelet should not try to get the container's log from the json log file (since it's not there) but should instead ask docker for the logs.

How to reproduce it (as minimally and precisely as possible):

  1. Configure dockerd to use the journald log driver. Add --log-driver=journald to its command line.
  2. Create a pod with a container that has terminationMessagePolicy set to FallbackToLogsOnError. For example, put the following into busybox-pod.yaml:
---
apiVersion: v1
kind: Pod
metadata:
  name: busybox-pod
spec:
  containers:
  - image: gcr.io/google_containers/busybox
    name: busybox
    terminationMessagePolicy: FallbackToLogsOnError
    command: ["/bin/sh"]
    args: ["-c", "echo Last line of container log; exit 1"]
  restartPolicy: Never

then run

kubectl create -f ~/busybox-pod.yaml
  1. Wait for the pod to exit with an error
  2. Look at the termination message:
kubectl get pod -o go-template="{{index .status.containerStatuses 0 "state" "terminated" "message"}}' busybox-pod

If you have reproduced the problem, you'll see something like the above error message. If it's working, you'll see

Last line of container log

Anything else we need to know?:
This is related to #42195 from a few months ago.

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.sig/nodeCategorizes an issue or PR as relevant to SIG Node.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions