Configured plugin systemd_logs does not exist

[evaldasou]

Hey Heptio Team.

When I check logs of my just deployed pod it exists with this error:

root:heptio# kubectl logs -f sonobuoy --namespace=heptio-sonobuoy
I0807 13:53:51.548619       1 loader.go:46] Scanning plugins in ./plugins.d (pwd: /)
I0807 13:53:51.548852       1 loader.go:46] Scanning plugins in /etc/sonobuoy/plugins.d (pwd: /)
I0807 13:53:51.548880       1 loader.go:46] Scanning plugins in ~/sonobuoy/plugins.d (pwd: /)
E0807 13:53:51.548907       1 master.go:49] Configured plugin systemd_logs does not exist

Any ideas , something is missing on the cluster itself?

[kensimon]

I think this is a race condition with how the resources in examples/quickstart are created... we create 2 configmaps, one with /etc/sonobuoy/config.json, and one with /etc/sonobuoy/plugins.d/*... it's possible the pod launched after the config.json was available but before the plugins.d files were.

If you do kubectl -n heptio-sonobuoy delete pod sonobuoy and re-run the kubectl apply -f ./examples/quickstart, does it work for you then?

[evaldasou]

thanks a lot @kensimon ! it worked this time after I have deleted pod!

[jamesbuckett]

I am still getting this issue : Configured plugin systemd_logs does not exist

I tried the work around as described above but no joy after two attempts.

Environment is :

heptio-sonobuoy Pods
No resources found, use --show-all to see completed objects.

heptio-sonobuoy Services
NAME              CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
sonobuoy-master   10.31.243.189   <none>        8080/TCP   57m

Nodes
NAME                                             STATUS    AGE       VERSION   EXTERNAL-IP      OS-IMAGE                             KERNEL-VERSION
gke-heptio-sonobuoy-default-pool-d1297397-7ztn   Ready     1h        v1.6.7    35.198.193.241   Container-Optimized OS from Google   4.4.52+
gke-heptio-sonobuoy-default-pool-d1297397-cwkf   Ready     1h        v1.6.7    35.198.210.119   Container-Optimized OS from Google   4.4.52+
gke-heptio-sonobuoy-default-pool-d1297397-k001   Ready     1h        v1.6.7    35.198.215.38    Container-Optimized OS from Google   4.4.52+

Error is :

 jamesb@Surface:~/sonobuoy$ kubectl logs -f sonobuoy --namespace=heptio-sonobuoy
I0818 08:09:05.243769       1 loader.go:47] Scanning plugins in ./plugins.d (pwd: /)
I0818 08:09:05.243991       1 loader.go:47] Scanning plugins in /etc/sonobuoy/plugins.d (pwd: /)
I0818 08:09:05.244059       1 loader.go:47] Scanning plugins in ~/sonobuoy/plugins.d (pwd: /)
E0818 08:09:05.244103       1 master.go:50] Configured plugin systemd_logs does not exist

[timothysc]

reopening, @kensimon could you please verify.

[chuckha]

I've seen this when I have an invalid configuration in my plugin. @jamesbuckett If you provide your plugin we can at least rule that out.

[gamename]

I'm seeing this issue as well. It happens about every third time I start a test.
My environment:

core@ip-10-223-11-108 ~ $ kubectl get svc --all-namespaces
NAMESPACE         NAME                   CLUSTER-IP       EXTERNAL-IP   PORT(S)                                    AGE
default           default-http-backend   10.233.211.83    <none>        80/TCP                                     4d
default           kubernetes             10.233.0.1       <none>        443/TCP                                    4d
heptio-sonobuoy   sonobuoy-master        10.233.18.18     <none>        8080/TCP                                   21m
kube-system       heapster               10.233.156.200   <none>        80/TCP                                     4d
kube-system       kubedns                10.233.0.3       <none>        53/UDP,53/TCP                              4d
kube-system       kubernetes-dashboard   10.233.134.41    <none>        9090/TCP                                   4d
kube-system       monitoring-grafana     10.233.150.152   <none>        80/TCP                                     4d
kube-system       monitoring-influxdb    10.233.161.142   <none>        8083/TCP,8086/TCP                          4d
kube-system       nginx-ingress-lb       10.233.138.40    <nodes>       80:31103/TCP,443:30755/TCP,442:30146/TCP   4d
core@ip-10-223-11-108 ~ $ kubectl get no
NAME                                          STATUS                     AGE       VERSION
ip-10-223-11-108.us-west-2.compute.internal   Ready,SchedulingDisabled   4d        v1.6.4+coreos.0
ip-10-223-11-70.us-west-2.compute.internal    Ready                      4d        v1.6.4+coreos.0
ip-10-223-3-100.us-west-2.compute.internal    Ready                      4d        v1.6.4+coreos.0
ip-10-223-3-106.us-west-2.compute.internal    Ready,SchedulingDisabled   4d        v1.6.4+coreos.0
ip-10-223-5-189.us-west-2.compute.internal    Ready                      4d        v1.6.4+coreos.0

The error

core@ip-10-223-11-108 ~ $ kubectl logs -f sonobuoy --namespace=heptio-sonobuoy
I0924 10:56:52.828142       1 loader.go:47] Scanning plugins in ./plugins.d (pwd: /)
I0924 10:56:52.828261       1 loader.go:47] Scanning plugins in /etc/sonobuoy/plugins.d (pwd: /)
I0924 10:56:52.828292       1 loader.go:47] Scanning plugins in ~/sonobuoy/plugins.d (pwd: /)
E0924 10:56:52.828366       1 master.go:50] Configured plugin systemd_logs does not exist

[timothysc]

Thx for reporting, I'll start investigating early this week.

[timothysc]

So I'll add more verbose logging to try and help diagnose, but I wonder if there was a config map mount race that exists in 1.6?

[timothysc]

So I'm traipsing through the control flow and nothing stands out atm. I'll have a PR to add more logging so we should get better visibility when the condition occurs, but I'm still leaning that this is a race condition on config map mounting.

[timothysc]

I'm going to untag this release and wait on feedback from testing. I've added more logging so if the issue occurs again we should have a better bead on whether it's a race condition or not.