Unexpected SIGTERM to dy-sidecar and dy-proxy pair

[GitHK]

It was observed that on autoscaled nodes, dy-sidecar and dy-proxy pair, sometimes get a SIGTERM out of the blue.

I am sure this is not initiated by the director-v2 since it would first try to save the state, push the outputs, before trying to remove both services.

For reference have a look at logs from 67fa84ec-5a78-49b8-8aca-0ce179d84021 AWS prod. It can clearly be seen that the sidecar and proxy receive a SIGERM and shut down.

[GitHK]

This was also observed again on a node with 3 dy-services running on them. All 3 pairs of dy-sidecar and dy-proxy received the SIGTERM at the same time.

The obvious culprit would be autoscaling draining the node too early.

[sanderegg]

@GitHK could you please add the command I should use in Graylog to find these cases? thanks.

[GitHK]

Related to this case. Currently on AWS staging unexpected behaviour. Containers are in an orphaned state. If we look at the details in the screenshots it's stated that the task belongs to a node which was deleted.

Unfortunately the longs are partial. If you filter by container_name: /.05827b71-86b1-48b4-9f5a-8ee5c4d545e8./ the only thing I see is that the sidecar was trying to shut down, but I don't see a lot of logs related to starting so I don't think these are that helpful.

@matusdrobuliak66 was also involved in this investigation.

[matusdrobuliak66]

Just not to forget, on 17.4. somewhere around that time we also turned off the autoscaling e2e tests (therefore this might be an outlier):

[sanderegg]

These services were manually deleted. @GitHK as I already asked. do you have a link to such an error?
If not I will unassign/close the case.

[GitHK]

We have been monitoring the staging deployment and there was not even a single incident regarding sidecars the last week. Let alone with autoscaled node. I'd say we can close the case for now.

[sanderegg]

Ok then I close it.