Batect Pre 0.80.0 compatibility with newer versions of Docker on Windows.

[Tobytee17]

Describe the bug

When using a pre-0.80.0 version of Batect on a Windows machine with newer versions of Docker Docker version 20.10.21, build baeda1f for example, Batect is unable to properly detect when tasks have completed and spin down relevant containers at the end of the task. Older versions of Docker, for example Docker version 19.03.12, build 0ed913b8- still work with pre-0.80.0 versions of Batect

When a task is completed, the container hangs until you press a few keys in the terminal, this is then logged:

During execution of step of kind 'RunContainerStep': batect.os.WindowsNativeMethodException: Invoking native method WriteFile failed with error ERROR_NO_DATA (The pipe is being closed).

Clean up: done

The task modules:vpc failed. See above for details.

Relevant piece of the Batect log generated by adding the --log-file argument to a task that hangs:

{"@timestamp":"2023-01-09T23:28:26.9071734Z","@message":"Unhandled exception during task step execution.","@severity":"error","@source":"batect.execution.ParallelExecutionManager","@processId":3524,"@threadId":36,"@threadName":"pool-1-thread-4","exception":"batect.os.WindowsNativeMethodException: Invoking native method WriteFile failed with error ERROR_NO_DATA (The pipe is being closed).\r\n\tat batect.os.NativeMethodsKt.throwWindowsNativeMethodFailed(NativeMethods.kt:55)\r\n\tat batect.os.NativeMethodsKt.throwWindowsNativeMethodFailed(NativeMethods.kt:48)\r\n\tat batect.sockets.namedpipes.NativeMethods.startWrite(NativeMethods.kt:156)\r\n\tat batect.sockets.namedpipes.NativeMethods.writeToNamedPipe(NativeMethods.kt:137)\r\n\tat batect.sockets.namedpipes.NamedPipe.write(NamedPipe.kt:25)\r\n\tat batect.sockets.namedpipes.NamedPipeSocket$outputStream$1.write(NamedPipeSocket.kt:78)\r\n\tat okio.OutputStreamSink.write(JvmOkio.kt:57)\r\n\tat okio.AsyncTimeout$sink$1.write(AsyncTimeout.kt:99)\r\n\tat okio.RealBufferedSink.close(RealBufferedSink.kt:279)\r\n\tat batect.docker.run.ContainerIOStreamer.streamWithConnectedOutput(ContainerIOStreamer.kt:60)\r\n\tat batect.docker.run.ContainerIOStreamer.stream(ContainerIOStreamer.kt:38)\r\n\tat batect.docker.client.ContainersClient.run(ContainersClient.kt:76)\r\n\tat batect.execution.model.steps.runners.RunContainerStepRunner.run(RunContainerStepRunner.kt:39)\r\n\tat batect.execution.model.steps.TaskStepRunner.run(TaskStepRunner.kt:40)\r\n\tat batect.execution.ParallelExecutionManager.runStep$lambda-3(ParallelExecutionManager.kt:151)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\r\n\tat java.base/java.lang.Thread.run(Thread.java:829)\r\n","step":{"type":"batect.execution.model.steps.RunContainerStep","container":"workspace","dockerContainer":{"id":"15e610cdd411f34b3b7607c00e4c2461d299d2eed0f3add4eb82188dfb77ca11","name":"tf-cra-aws-workspace-o3xaso"}}}
{"@timestamp":"2023-01-09T23:28:26.9092703Z","@message":"Event received.","@severity":"info","@source":"batect.execution.TaskStateMachine","@processId":3524,"@threadId":36,"@threadName":"pool-1-thread-4","event":{"type":"batect.execution.model.events.ExecutionFailedEvent","message":"During execution of step of kind 'RunContainerStep': batect.os.WindowsNativeMethodException: Invoking native method WriteFile failed with error ERROR_NO_DATA (The pipe is being closed)."}}
{"@timestamp":"2023-01-09T23:28:26.9098814Z","@message":"Trying to get next step to execute.","@severity":"info","@source":"batect.execution.TaskStateMachine","@processId":3524,"@threadId":36,"@threadName":"pool-1-thread-4","currentStage":"batect.execution.model.stages.RunStage@74170687","stepsStillRunning":true}
{"@timestamp":"2023-01-09T23:28:26.9116206Z","@message":"Task has failed, not returning any further work while existing work completes.","@severity":"info","@source":"batect.execution.TaskStateMachine","@processId":3524,"@threadId":36,"@threadName":"pool-1-thread-4"}
{"@timestamp":"2023-01-09T23:28:26.9116206Z","@message":"Trying to get next step to execute.","@severity":"info","@source":"batect.execution.TaskStateMachine","@processId":3524,"@threadId":36,"@threadName":"pool-1-thread-4","currentStage":"batect.execution.model.stages.RunStage@74170687","stepsStillRunning":false}
{"@timestamp":"2023-01-09T23:28:26.9121604Z","@message":"Task has failed and existing work has finished. Beginning cleanup.","@severity":"info","@source":"batect.execution.TaskStateMachine","@processId":3524,"@threadId":36,"@threadName":"pool-1-thread-4"}
{"@timestamp":"2023-01-09T23:28:26.9127142Z","@message":"Created cleanup stage.","@severity":"info","@source":"batect.execution.model.stages.CleanupStagePlanner","@processId":3524,"@threadId":36,"@threadName":"pool-1-thread-4","manualCleanupCommands":["docker rm --force --volumes 15e610cdd411f34b3b7607c00e4c2461d299d2eed0f3add4eb82188dfb77ca11","docker network rm b04c278fbb46df9b4af1e0432040b4dd54afd97832aeee5ed078d068e404fe84"],"rules":[{"type":"batect.execution.model.rules.cleanup.DeleteTaskNetworkStepRule","network":{"id":"b04c278fbb46df9b4af1e0432040b4dd54afd97832aeee5ed078d068e404fe84"},"containersThatMustBeRemovedFirst":["workspace"],"manualCleanupCommand":"docker network rm b04c278fbb46df9b4af1e0432040b4dd54afd97832aeee5ed078d068e404fe84"},{"type":"batect.execution.model.rules.cleanup.StopContainerStepRule","container":"workspace","dockerContainer":{"id":"15e610cdd411f34b3b7607c00e4c2461d299d2eed0f3add4eb82188dfb77ca11","name":"tf-cra-aws-workspace-o3xaso"},"containersThatMustBeStoppedFirst":[],"manualCleanupCommand":null},{"type":"batect.execution.model.rules.cleanup.RemoveContainerStepRule","container":"workspace","dockerContainer":{"id":"15e610cdd411f34b3b7607c00e4c2461d299d2eed0f3add4eb82188dfb77ca11","name":"tf-cra-aws-workspace-o3xaso"},"containerWasStarted":true,"manualCleanupCommand":"docker rm --force --volumes 15e610cdd411f34b3b7607c00e4c2461d299d2eed0f3add4eb82188dfb77ca11"}],"pastEvents":[{"type":"batect.execution.model.events.TaskNetworkCreatedEvent","network":{"id":"b04c278fbb46df9b4af1e0432040b4dd54afd97832aeee5ed078d068e404fe84"}},{"type":"batect.execution.model.events.ImageBuiltEvent","container":"workspace","image":{"id":"sha256:14fd3ff84080788c34723204ef2ff95df72d67d3401b757c58d7a0d8bfc99bce"}},{"type":"batect.execution.model.events.ContainerCreatedEvent","container":"workspace","dockerContainer":{"id":"15e610cdd411f34b3b7607c00e4c2461d299d2eed0f3add4eb82188dfb77ca11","name":"tf-cra-aws-workspace-o3xaso"}},{"type":"batect.execution.model.events.ContainerStartedEvent","container":"workspace"},{"type":"batect.execution.model.events.ContainerBecameHealthyEvent","container":"workspace"},{"type":"batect.execution.model.events.ContainerBecameReadyEvent","container":"workspace"},{"type":"batect.execution.model.events.ExecutionFailedEvent","message":"During execution of step of kind 'RunContainerStep': batect.os.WindowsNativeMethodException: Invoking native method WriteFile failed with error ERROR_NO_DATA (The pipe is being closed)."}]}
{"@timestamp":"2023-01-09T23:28:26.914722Z","@message":"Returning first step from cleanup stage.","@severity":"info","@source":"batect.execution.TaskStateMachine","@processId":3524,"@threadId":36,"@threadName":"pool-1-thread-4"}
{"@timestamp":"2023-01-09T23:28:26.914722Z","@message":"Trying to get next step to execute.","@severity":"info","@source":"batect.execution.TaskStateMachine","@processId":3524,"@threadId":36,"@threadName":"pool-1-thread-4","currentStage":"batect.execution.model.stages.CleanupStage@56d83595","stepsStillRunning":false}
{"@timestamp":"2023-01-09T23:28:26.9157819Z","@message":"Step is ready to execute.","@severity":"info","@source":"batect.execution.TaskStateMachine","@processId":3524,"@threadId":36,"@threadName":"pool-1-thread-4","step":{"type":"batect.execution.model.steps.StopContainerStep","container":"workspace","dockerContainer":{"id":"15e610cdd411f34b3b7607c00e4c2461d299d2eed0f3add4eb82188dfb77ca11","name":"tf-cra-aws-workspace-o3xaso"}}}
{"@timestamp":"2023-01-09T23:28:26.9177244Z","@message":"Trying to get next step to execute.","@severity":"info","@source":"batect.execution.TaskStateMachine","@processId":3524,"@threadId":36,"@threadName":"pool-1-thread-4","currentStage":"batect.execution.model.stages.CleanupStage@56d83595","stepsStillRunning":true}
{"@timestamp":"2023-01-09T23:28:26.9177244Z","@message":"Running step.","@severity":"info","@source":"batect.execution.ParallelExecutionManager","@processId":3524,"@threadId":43,"@threadName":"pool-1-thread-7","step":{"type":"batect.execution.model.steps.StopContainerStep","container":"workspace","dockerContainer":{"id":"15e610cdd411f34b3b7607c00e4c2461d299d2eed0f3add4eb82188dfb77ca11","name":"tf-cra-aws-workspace-o3xaso"}}}
{"@timestamp":"2023-01-09T23:28:26.918719Z","@message":"No steps ready to execute.","@severity":"info","@source":"batect.execution.TaskStateMachine","@processId":3524,"@threadId":36,"@threadName":"pool-1-thread-4"}
{"@timestamp":"2023-01-09T23:28:26.9217125Z","@message":"Stopping container.","@severity":"info","@source":"batect.docker.api.ContainersAPI","@processId":3524,"@threadId":43,"@threadName":"pool-1-thread-7","container":{"id":"15e610cdd411f34b3b7607c00e4c2461d299d2eed0f3add4eb82188dfb77ca11","name":"tf-cra-aws-workspace-o3xaso"}}
{"@timestamp":"2023-01-09T23:28:26.9227515Z","@message":"HTTP request starting.","@severity":"debug","@source":"batect.logging.HttpLoggingInterceptor","@processId":3524,"@threadId":43,"@threadName":"pool-1-thread-7","method":"POST","contentLength":0,"contentType":"text/plain; charset=utf-8","url":"http://f267e762-c464-4a4e-8987-a92b8bb1fe5d.namedpipe/v1.37/containers/15e610cdd411f34b3b7607c00e4c2461d299d2eed0f3add4eb82188dfb77ca11/stop?timeout=10"}
{"@timestamp":"2023-01-09T23:28:26.9632262Z","@message":"HTTP response received.","@severity":"debug","@source":"batect.logging.HttpLoggingInterceptor","@processId":3524,"@threadId":43,"@threadName":"pool-1-thread-7","code":304,"method":"POST","contentLength":0,"message":"Not Modified","contentType":null,"url":"http://f267e762-c464-4a4e-8987-a92b8bb1fe5d.namedpipe/v1.37/containers/15e610cdd411f34b3b7607c00e4c2461d299d2eed0f3add4eb82188dfb77ca11/stop?timeout=10"}
{"@timestamp":"2023-01-09T23:28:26.9650032Z","@message":"Container has already stopped.","@severity":"warning","@source":"batect.docker.api.ContainersAPI","@processId":3524,"@threadId":43,"@threadName":"pool-1-thread-7"}

The issue #1335 prevents us from upgrading the version of Batect used in a number of projects (as our GitLab build system uses Alpine based containers as their base image).

Migrating projects to new runners using a different flavor of Linux as it's base image is definitely an option, but I wanted to raise this ticket first to confirm this is a compatibility issue, and to check whether putting in a fix into the latest pre-0.80.0 version of Batect would be a possibility (assuming it didn't require a massive amount of work).

Reproduction repo

No response

Steps to reproduce

Steps below must be completed on a windows machine. This same scenario works when using WSL2 (Ubuntu 20.04 LTS) and on machines running linux (Oracle Linux Server 8.7 for example).

1. Install Docker Desktop, and ensure you are using a relatively new version of Docker, for example Docker version 20.10.21, build baeda1f
2. Install a pre-0.80.0 version of Batect, from example 0.79.1
3. Run any task.
4. Wait until you know the task has completed, observe Batect hanging, tap a few keys while in the terminal and observe the logged error.

Expected behaviour

The task runs to completion and Batect teardown the container

Actual behaviour

The command inside the task completes, then the task hangs indefinitely requiring input from the user to trigger the completion/error.

Version information

Batect version:    0.79.1
Built:             2022-04-24 05:04:00 +0000
Built from commit: a33d32f6303e3905e6fe57178dc4f4dadc3ab15d (commit date: 2022-04-24 14:28:58 +1000)
JVM version:       Oracle Corporation OpenJDK 64-Bit Server VM 11.0.12
OS version:        Windows 10 10.0 amd64
Docker version:    20.10.21 (API version: 1.41, minimum supported API version: 1.12, commit: 3056208, operating system: 'linux', experimental: false)
Git version:       2.39.1.windows.1

For documentation and further information on Batect, visit https://github.com/batect/batect.

Version 0.83.2 of Batect is now available (you have 0.79.1).
To upgrade to the latest version, run './batect --upgrade'.
For more information, visit https://github.com/batect/batect/releases/tag/0.83.2.

Any other information

No response

[charleskorn]

Hi @Tobytee17, I only support the most recent version of Batect unfortunately.