Reopen Docker event stream when getting EOF error

[fenxiong]

Summary

Fix #591.

Previously, agent did not reopen docker event stream when it gets EOF/UnexpectedEOF error. When fsouza/go-dockerclient was used, we had this behavior because fsouza/go-dockerclient can't reopen the stream upon EOF/UnexpectedEOF due to some of its own implementation details (see fsouza/go-dockerclient#171) and we did not retry in our layer. When migrated to docker/docker/client, this behavior was maintained.

In the case that Docker is restarted (and container live-restore is enabled), agent's event stream listening goroutine gets an io.UnexpectedEOF and finishes, and we never open the stream again and as a result loss all Docker events until the agent is restarted. So making this changes to reopen Docker event stream when getting EOF error.

Implementation details

Few changes in ContainerEvents.

Testing

Updated the TestContainerEventsEOFError to conform with new behavior.

Manually tested with the steps to reproduce described in #591 and verified that it's fixed.

Description for the changelog

Fixed a bug where when Docker container live restore is enabled, agent stops listening to Docker container event after Docker restarts #2240.

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[shubham2892]

Is error io.UnexpectedEOF only expected when docker restarts, are there any other cases when we get the same error, that we need to handle differently?

[fenxiong]

Is error io.UnexpectedEOF only expected when docker restarts, are there any other cases when we get the same error, that we need to handle differently?

It can happen as long as the connection to docker socket is closed. IMO the agent should always attempt to reopen Docker event stream, since we rely on that to detect changes from customer's container. I can't think of a situation where we want the agent to stop listening to Docker events other than the case that the task engine is shut down - in that case the context will be canceled and we will stop listening.

[adnxn]

changes lgtm, non blocking but would be nice to test more exhaustively as i mentioned offline.

[fierlion]

These changes look good to me.