Include journald in compilation path to log to systemd journal

[grassdionera]

What happened?

switching to new packaging pkgs.k8s.io with Ubuntu 22.04 and having log_to_journald set to true breaks all containers with the errors

Mar 16 11:32:24 crio-conmon[3397]: conmon 2147ca5a428c105965f7 <error>: Include journald in compilation path to log to systemd journal
Mar 16 11:32:24 crio[1094]: time="2024-03-16 11:32:24.972446790+01:00" level=debug msg="Response error: failed to wait exit status 1 after failing with: write child: broken pipe" file="otel-collector/interceptors.go:71" id=a2e89013-85ba-45d5-815e-3ca72e3e4769 name=/runtime.v1.RuntimeService/CreateContainer

all containers, are Stuck in CreateContainerErrror

What did you expect to happen?

Container should be created normaly like with the old opensuse Packaging and logging to journal should be possible

How can we reproduce it (as minimally and precisely as possible)?

• setup k8s cluster with kubeadm 1.29 on Ubuntu 22.04
• setup crio 1.28.2 with the old packaging guide <1.29 and opensuse Deb repo
• set log_to_journald=true
• add new k8s.pkgs.io addon repo to apt source list
• update crio to latest 1.29 version from new repo
• restart crio and kubelet

Anything else we need to know?

we testet all version of crio within the new packaging, and it happens to all of them
we removed the old dependencies conmon, containers-common, libgpgme11 and cri-o-runc and also tried a full reinstall. but nothing changed

CRI-O and Kubernetes version

$ crio --version
crio version 1.29.2
Version:        1.29.2
GitCommit:      d317b5dc918bbfbc78481072a0d93e572aa8d0e8
GitCommitDate:  2024-02-22T19:23:38Z
GitTreeState:   clean
BuildDate:      1970-01-01T00:00:00Z
GoVersion:      go1.21.1
Compiler:       gc
Platform:       linux/amd64
Linkmode:       static
BuildTags:
  static
  netgo
  osusergo
  exclude_graphdriver_btrfs
  exclude_graphdriver_devicemapper
  seccomp
  apparmor
  selinux
LDFlags:          unknown
SeccompEnabled:   true
AppArmorEnabled:  true

$ kubectl version --output=json
{
  "clientVersion": {
    "major": "1",
    "minor": "29",
    "gitVersion": "v1.29.2",
    "gitCommit": "4b8e819355d791d96b7e9d9efe4cbafae2311c88",
    "gitTreeState": "clean",
    "buildDate": "2024-02-14T10:40:49Z",
    "goVersion": "go1.21.7",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "kustomizeVersion": "v5.0.4-0.20230601165947-6ce0bf390ce3",
  "serverVersion": {
    "major": "1",
    "minor": "29",
    "gitVersion": "v1.29.2",
    "gitCommit": "4b8e819355d791d96b7e9d9efe4cbafae2311c88",
    "gitTreeState": "clean",
    "buildDate": "2024-02-14T10:32:40Z",
    "goVersion": "go1.21.7",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}

OS version

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
$ uname -a
Linux master1.local.integration.dionera.dev 5.15.0-100-generic #110-Ubuntu SMP Wed Feb 7 13:27:48 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Additional environment details (AWS, VirtualBox, physical, etc.)

barebone physical

[kwilczynski]

@grassdionera, thank you for getting in touch! Sorry, you are having issues.

Do you have a version with which logging to journald works for you? If so, then which one was the last working one?

This most likely is a regression. As such, I am trying to establish with which release it was introduced.

[grassdionera]

it was working with the version 1.28.2~0 from
https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/1.28/xUbuntu_22.04

[haircommander]

yeah this would be a difference in the conmon version. when we build conmon statically we don't include the journald libraries in. are you able to use the conmon from kubic for now? you may need to edit the crio configuration to point to the right conmon.

[grassdionera]

yes installing the conmon opensuse kubic version from 1.28 repo and setting the config to

[crio.runtime]
log_to_journald=true
conmon="/bin/conmon"

is working again as expected and is a good temporäry workaround. thanks.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[grassdionera]

will you change your builds to include the journald libraries?

[kwilczynski]

/remove-lifecycle stale

[kwilczynski]

/assign haircommander
/assign saschagrunert

[saschagrunert]

That's a conmon build thing: https://github.com/containers/conmon/blob/e21e7c85b7637e622f21c57675bf1154fc8b1866/src/ctr_logging.c#L134-L136

We have no support for systemd in conmon right now.