drop-in directory is ignored

[gfiedler]

Manpage of systemd.unit describes an optional drop-in directory for systemd unit files:

Along with a unit file foo.service, a "drop-in" directory foo.service.d/ may exist.

This files are ignored.

An example is Kubernetes, which installs files in drop-in directories.

curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install kubelet kubeadm

installs the following files:

/lib/systemd/system/kubelet.service
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf

The command

systemctl.py start kubelet

doesn't take 10-kubeadm.conf into account and kubelet service starts with wrong parameters.

[gfiedler]

I find out that there is already support for this feature implemented here. But it doesn't work with the kubadm drop-in for any reason.

docker-systemctl-replacement v1.4.2456

[gdraheim]

Yes, the feature is implemented and I have seen it to work properly.

So it would be interesting to see which value is considered to be wrong. Would you mind to attach the relevant files? (may be with passwords xxxed). Any idea what values should be different?

Also check the output of the properties like

systemctl.py environment cron
systemctl.py show cron
systemctl.py cat cron

[gdraheim]

By the way, I do see now that there is a different directory prefix. Would you mind moving the config file into a different path like

/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

[gfiedler]

# systemctl.py cat kubelet
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/home/

[Service]
ExecStart=/usr/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target

This is exactly the content of /lib/systemd/system/kubelet.service without the overrides of 10-kubeadm.conf.

# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

[gfiedler]

Moving to /lib/systemd/system/kubelet.service.d/10-kubeadm.conf doesn't work either.

[gdraheim]

I have started to check into that

1. only in /lib/ the *.confs are actually read
2. even after moving, the values are not added

So, it is actually two problems

• a plain bug about reading the confs where "*.conf" is parsed but the values do not get present
• systemd' systemctl behaves different by adding overrides from all locations

[gdraheim]

Actually, my testcases war partially wrong. The systemd config reader is just fine, but I noticed from your example that is was missing the option to reset a value by giving an emtpy line in the xx.service descriptor.

That is also tested in test_8051 now. The test_8052 will check your specific kubelet example, so it should work in your setup as expected by now - at version v1.4.2505

[gfiedler]

I tested it with version v1.4.2505 and it worked like a charme.

There should be more people like you. Thank you for the great support!

[gdraheim]

I have started to create a number of testcases to cover the drop-in functionalty..... however try-and-error did show that systemd will check all systemd directories but it will ignore some drop-in files.

It seems that the drop-in files are loaded alphabetically by filename, and if the filename exists in multiple override.d directories then only the uppermost is used, mostly /etc stuff.

That did require a complete reimplementation as for v1.4.2521

[gdraheim]

I guess it is fixed now.