Include logs from previous Pod incarnations in diagnostics

[pebrc]

Sorry for too many requests but do we have the way to implement things like previous logs?

kubectl logs <pod name> --previous

Sometimes ES nodes just left and we can't take the logs by kubectl logs but we have to add --previous separately.

Originally posted by @kunisen in #28 (comment)

[pebrc]

I did not realise this at the time but only found out about this now: the --previous flag in kubectl logs does not what I thought it would do. It does not stream the logs of previous incarnations of a given Pod but it would give you the logs of previous instances of a container in a Pod.

We almost never have the case were we have a single container inside a Pod restarting (unless there is something wrong in which case the previous container logs might be interesting). What we constantly have is Pods being deleting in order to roll out upgrades in a cluster. These logs would be interesting because the show for example what the Elasticsearch cluster was doing before an upgrade. However we have not means of retrieving the logs of already deleted Pods (even if a new Pod with the exact same name was created right after)

[kunisen]

However we have not means of retrieving the logs of already deleted Pods (even if a new Pod with the exact same name was created right after)

Thanks.
I think we may need to login to VM and grab the logs for those previously run pods. (not sure if the logs will be deleted or not)
Maybe the only way is to enable sidecar logging and use filebeat to send it to a central place for troubleshooting purpose.

But in either case, not a ECK diagnostics can do here so please let me close this ticket.

Thanks!

[stefnestor]

👋🏽 @kunisen @pebrc , may I confirm if this FR would answer a common high severity situation:

Users may experience ES bootlooping errors where Elasticsearch is cyclically exiting and the operator logs only report Elasticsearch cannot be reached yet, re-queuing. In this situation, Support asks user to send the pod's previous logs (ex online doc which uses --previous) in hopes they can gather the failed start-up logs. I may be misunderstanding, but this appears to be helpful when pods are listed in PENDING with READY like 0/1.

Secondly, Support internal KB id: 05f99124 lists this ballpark helpful when troubleshooting OOM, FWIW.

[pebrc]

@stefnestor I think you are right that for a repeatedly restarting container the previous logs might be helpful because the diagnostic tool might be running at a time when the container is currently not running due to backoff.