Docker log rotation stops after reaching limit on Windows nodes

[jsturtevant]

Describe the bug
After the limit for a containers logs are met as set up in #2478 the logs stop being collected. The container continues to run but no new logs are appended.

Steps To Reproduce

Create a windows cluster and deploy a pod that logs to much. Call kubetctl logs <pod> and look at time stamp of last record. Wait a few mins and call kubectl logs <pod> and call kubetctl logs <pod> again and the time stamp will be the same but should be just a few seconds old.

Expected behavior
Log rotation behaves as configured.

AKS Engine version
latest

Kubernetes version
all

Additional context
The fix is in the 19.3.09 release of docker. The latest Windows Server EE is at 19.3.05. We currently use Docker EE on windows nodes.

[jsturtevant]

A few options for mitigation while the next EE release is pending:

• A temporary mitigation would be to restart the deployment kubectl rollout restart deployment <deployment-name>. The pods would need to be restarted anytime the log file hits 50mb.

• workaround could also set the daemon.json to max-file of 1 on each node. The logs will continue to flow but they might miss logs while the log file gets overwritten. Optionally could make log file size bigger.

• Another option is to use Containerd, I have not tested to see if this resolved the issue.