Improve fluentd liveness probe

[vsinghal13]

Description

Add new liveness probe to fluentd deployment.

httpGet:
    path: /fluentd.pod.healthcheck?json=%7B%22log%22%3A+%22health+check%22%7D
    port: 9880

The rationale is that if Fluentd can accept log messages, it must be healthy.
The endpoint itself results in a new fluentd tag fluentd.pod-healthcheck
The query parameter in the URL defines a URL-encoded JSON object that looks like this:
{"log": "health check"}

Testing performed

• ci/build.sh
• Redeploy fluentd and fluentd-events pods
• Confirm events, logs, and metrics are coming in

[rvmiller89]

I'm confused why only logs.conf gets this liveness probe. Should this be moved to common.conf ?

[rvmiller89]

this seems very generous, do we need to wait 5 minutes before starting the liveness probe?

[vsinghal13]

Currently we do have wait of 5 minutes, so I kept that. It is recommended to be generous in that otherwise the pod might end up in infinite loop of being restarted during startup.

[samjsong]

5min does sound a little high, but the readiness probe will fail first and the pod will stop accepting data, and if we need more replicas in between that time then with autoscaler enabled I think we'll be okay. So I'm okay with leaving it at 5min

[frankreno]

LGTM, but any reason we would not also want to do this on the event deployment? The same logic holds true I believe.

[rvmiller89]

+1 for doing on events deployment, missed that part.