sonobuoy status doesn't report the tests are completed when the logs do.

[ceridwen]

What steps did you take and what happened:

I have automation running sonobuoy that has been using the logs to determine when the tests are finished and the results are available. When I changed it to use sonobuoy status, I've encountered a case where sonobuoy status thinks the tests are still running when the logs indicate they're finished:

$ ./sonobuoy status
PLUGIN		STATUS		COUNT
e2e		running		1
systemd_logs	complete	5

Sonobuoy is still running. Runs can take up to 60 minutes.

$ kubectl logs -n heptio-sonobuoy sonobuoy -f
time="2018-07-17T17:48:10Z" level=info msg="Scanning plugins in ./plugins.d (pwd: /)"
time="2018-07-17T17:48:10Z" level=info msg="Scanning plugins in /etc/sonobuoy/plugins.d (pwd: /)"
time="2018-07-17T17:48:10Z" level=info msg="Directory (/etc/sonobuoy/plugins.d) does not exist"
time="2018-07-17T17:48:10Z" level=info msg="Scanning plugins in ~/sonobuoy/plugins.d (pwd: /)"
time="2018-07-17T17:48:10Z" level=info msg="Directory (~/sonobuoy/plugins.d) does not exist"
time="2018-07-17T17:48:10Z" level=info msg="Filtering namespaces based on the following regex:.*|heptio-sonobuoy"
time="2018-07-17T17:48:10Z" level=info msg="Namespace default Matched=true"
time="2018-07-17T17:48:10Z" level=info msg="Namespace e2e-tests-nettest-5qgxw Matched=true"
time="2018-07-17T17:48:10Z" level=info msg="Namespace heptio-sonobuoy Matched=true"
time="2018-07-17T17:48:10Z" level=info msg="Namespace kube-public Matched=true"
time="2018-07-17T17:48:10Z" level=info msg="Namespace kube-system Matched=true"
time="2018-07-17T17:48:10Z" level=info msg="Starting server Expected Results: [{ e2e} {fab10-compute-1 systemd_logs} {fab10-compute-2 systemd_logs} {fab10-compute-3 systemd_logs} {fab10-compute-4 systemd_logs} {fab10-compute-5 systemd_logs}]"
time="2018-07-17T17:48:10Z" level=info msg="starting aggregation server" address=0.0.0.0 port=8080
time="2018-07-17T17:48:10Z" level=info msg="Running plugin" plugin=e2e
time="2018-07-17T17:48:10Z" level=info msg="Running plugin" plugin=systemd-logs
time="2018-07-17T17:48:13Z" level=info msg="received aggregator request" client_cert=systemd-logs node=fab10-compute-1 plugin_name=systemd_logs
time="2018-07-17T17:48:13Z" level=info msg="received aggregator request" client_cert=systemd-logs node=fab10-compute-3 plugin_name=systemd_logs
time="2018-07-17T17:48:13Z" level=info msg="received aggregator request" client_cert=systemd-logs node=fab10-compute-2 plugin_name=systemd_logs
time="2018-07-17T17:48:13Z" level=info msg="received aggregator request" client_cert=systemd-logs node=fab10-compute-5 plugin_name=systemd_logs
time="2018-07-17T17:48:13Z" level=info msg="received aggregator request" client_cert=systemd-logs node=fab10-compute-4 plugin_name=systemd_logs
time="2018-07-17T17:48:30Z" level=info msg="received aggregator request" client_cert=e2e plugin_name=e2e
time="2018-07-17T17:48:30Z" level=info msg="Running non-ns query"
time="2018-07-17T17:48:30Z" level=error msg="error querying CustomResourceDefinitions: the server could not find the requested resource"
time="2018-07-17T17:48:30Z" level=info msg="Collecting Node Configuration and Health..."
time="2018-07-17T17:48:30Z" level=info msg="Creating host results for fab10-compute-1 under /tmp/sonobuoy/d6b2dd57-773c-4823-8e3c-69fc80a4b137/hosts/fab10-compute-1\n"
time="2018-07-17T17:48:30Z" level=warning msg="Could not get configz endpoint for node fab10-compute-1: the server could not find the requested resource"
time="2018-07-17T17:48:30Z" level=warning msg="Could not get healthz endpoint for node fab10-compute-1: the server could not find the requested resource"
time="2018-07-17T17:48:30Z" level=info msg="Creating host results for fab10-compute-2 under /tmp/sonobuoy/d6b2dd57-773c-4823-8e3c-69fc80a4b137/hosts/fab10-compute-2\n"
time="2018-07-17T17:48:30Z" level=warning msg="Could not get configz endpoint for node fab10-compute-2: the server could not find the requested resource"
time="2018-07-17T17:48:30Z" level=warning msg="Could not get healthz endpoint for node fab10-compute-2: the server could not find the requested resource"
time="2018-07-17T17:48:30Z" level=info msg="Creating host results for fab10-compute-3 under /tmp/sonobuoy/d6b2dd57-773c-4823-8e3c-69fc80a4b137/hosts/fab10-compute-3\n"
time="2018-07-17T17:48:30Z" level=warning msg="Could not get configz endpoint for node fab10-compute-3: the server could not find the requested resource"
time="2018-07-17T17:48:30Z" level=warning msg="Could not get healthz endpoint for node fab10-compute-3: the server could not find the requested resource"
time="2018-07-17T17:48:30Z" level=info msg="Creating host results for fab10-compute-4 under /tmp/sonobuoy/d6b2dd57-773c-4823-8e3c-69fc80a4b137/hosts/fab10-compute-4\n"
time="2018-07-17T17:48:30Z" level=warning msg="Could not get configz endpoint for node fab10-compute-4: the server could not find the requested resource"
time="2018-07-17T17:48:31Z" level=warning msg="Could not get healthz endpoint for node fab10-compute-4: the server could not find the requested resource"
time="2018-07-17T17:48:31Z" level=info msg="Creating host results for fab10-compute-5 under /tmp/sonobuoy/d6b2dd57-773c-4823-8e3c-69fc80a4b137/hosts/fab10-compute-5\n"
time="2018-07-17T17:48:31Z" level=warning msg="Could not get configz endpoint for node fab10-compute-5: the server could not find the requested resource"
time="2018-07-17T17:48:31Z" level=warning msg="Could not get healthz endpoint for node fab10-compute-5: the server could not find the requested resource"
time="2018-07-17T17:48:31Z" level=error msg="error querying ThirdPartyResources: don't know how to handle non-namespaced resource ThirdPartyResources"
time="2018-07-17T17:48:31Z" level=info msg="Running ns query (default)"
time="2018-07-17T17:48:33Z" level=info msg="Collecting Pod Logs..."
time="2018-07-17T17:48:33Z" level=error msg="error querying PodPresets: the server could not find the requested resource (get podpresets.settings.k8s.io)"
time="2018-07-17T17:48:35Z" level=info msg="Running ns query (e2e-tests-nettest-5qgxw)"
time="2018-07-17T17:48:36Z" level=info msg="Collecting Pod Logs..."
time="2018-07-17T17:48:37Z" level=error msg="error querying PodPresets: the server could not find the requested resource (get podpresets.settings.k8s.io)"
time="2018-07-17T17:48:38Z" level=info msg="Running ns query (heptio-sonobuoy)"
time="2018-07-17T17:48:39Z" level=info msg="Collecting Pod Logs..."
time="2018-07-17T17:48:42Z" level=error msg="error querying PodPresets: the server could not find the requested resource (get podpresets.settings.k8s.io)"
time="2018-07-17T17:48:44Z" level=info msg="Running ns query (kube-public)"
time="2018-07-17T17:48:45Z" level=info msg="Collecting Pod Logs..."
time="2018-07-17T17:48:45Z" level=error msg="error querying PodPresets: the server could not find the requested resource (get podpresets.settings.k8s.io)"
time="2018-07-17T17:48:46Z" level=info msg="Running ns query (kube-system)"
time="2018-07-17T17:48:48Z" level=info msg="Collecting Pod Logs..."
time="2018-07-17T17:49:11Z" level=error msg="error querying PodPresets: the server could not find the requested resource (get podpresets.settings.k8s.io)"
time="2018-07-17T17:49:13Z" level=info msg="Results available at /tmp/sonobuoy/201807171748_sonobuoy_d6b2dd57-773c-4823-8e3c-69fc80a4b137.tar.gz"
time="2018-07-17T17:49:13Z" level=info msg="no-exit was specified, sonobuoy is now blocking"

What did you expect to happen:

sonobuoy status to show the tests have completed.

Environment:

• Sonobuoy version: 0.11.3
• Kubernetes version: (use kubectl version): 1.10.5
• Kubernetes installer & version: kubespray
• Cloud provider or hardware configuration: bare metal
• OS (e.g. from /etc/os-release): Ubuntu 16.04
• Sonobuoy tarball (which contains * below)
  201807171748_sonobuoy_d6b2dd57-773c-4823-8e3c-69fc80a4b137.tar.gz

[timothysc]

There are two parts.

1 - Check races on worker cleanup
2 - Aggregator done = clear status.