Intermittent Test Failures with Cosmos DB Emulator in Azure Pipelines using TestContainers

[noh626]

Hi,

I am trying to integrate TestContainers into my integration tests, which use Cosmos DB. While everything works fine locally, the tests often fail unexpectedly in the Azure pipeline.

This is my setup to run the Cosmos DB emulator container:

private readonly CosmosDbContainer container = new CosmosDbBuilder()
    .WithImage("mcr.microsoft.com/cosmosdb/windows/azure-cosmos-emulator")
    .WithName("IntegrationCosmosDB")
    .WithPortBinding(8081, 8081)
    .WithPortBinding(10251, 10251)
    .WithPortBinding(10252, 10252)
    .WithPortBinding(10253, 10253)
    .WithPortBinding(10254, 10254)
    .WithPortBinding(10255, 10255)
    .WithEnvironment("AZURE_COSMOS_EMULATOR_PARTITION_COUNT", "3")
    .WithEnvironment("AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE", "127.0.0.1")
    .WithEnvironment("AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE", "false")
    .WithWaitStrategy(Wait.ForWindowsContainer().UntilPortIsAvailable(8081))
    .WithCreateParameterModifier(parameters =>
    {
        parameters.Tty = true;
        parameters.HostConfig.Isolation = "hyperv";
    })
    .Build();

I'm using Hyper-V isolation mode since the host agent is Windows Server 2022, and the Cosmos DB emulator is not compatible with it. However, I frequently encounter the following error:

[testcontainers.org 00:00:00.23] Docker container 2d2ceb17cfcc created
[testcontainers.org 00:00:00.26] Start Docker container 2d2ceb17cfcc
[testcontainers.org 00:00:13.55] Wait for Docker container 2d2ceb17cfcc to complete readiness checks
[testcontainers.org 00:00:13.55] Execute "PowerShell -Command Exit(-Not((Test-NetConnection -ComputerName 'localhost' -Port 8081).TcpTestSucceeded))" at Docker container 2d2ceb17cfcc
[testcontainers.org 00:00:28.98] Execute "PowerShell -Command Exit(-Not((Test-NetConnection -ComputerName 'localhost' -Port 8081).TcpTestSucceeded))" at Docker container 2d2ceb17cfcc
[testcontainers.org 00:00:39.05] Execute "PowerShell -Command Exit(-Not((Test-NetConnection -ComputerName 'localhost' -Port 8081).TcpTestSucceeded))" at Docker container 2d2ceb17cfcc
[xUnit.net 00:00:39.94] Test.Integration.TestSomething [FAIL]
[testcontainers.org 00:00:39.07] Delete Docker container 2d2ceb17cfcc
Failed Test.Integration.TestSomething [1 ms]
Error Message:
Docker.DotNet.DockerApiException : Docker API responded with status code=Conflict, response={"message":"Container 2d2ceb17cfcc78d635c0ae00c2ae250a60912715323deefeaaab8e7faf4fab80 is not running"}
Stack Trace:
at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable1 handlers)
at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary2 headers, TimeSpan timeout, CancellationToken token)
at Docker.DotNet.ExecOperations.ExecCreateContainerAsync(String id, ContainerExecCreateParameters parameters, CancellationToken cancellationToken)
at DotNet.Testcontainers.Clients.DockerContainerOperations.ExecAsync(String id, IList1 command, CancellationToken ct) in //src/Testcontainers/Clients/DockerContainerOperations.cs:line 150
at DotNet.Testcontainers.Configurations.UntilUnixCommandIsCompleted.UntilAsync(IContainer container) in //src/Testcontainers/Configurations/WaitStrategies/UntilUnixCommandIsCompleted.cs:line 22
at DotNet.Testcontainers.Containers.DockerContainer.<>c__DisplayClass72_0.<g__CheckWaitStrategyAsync|1>d.MoveNext() in //src/Testcontainers/Containers/DockerContainer.cs:line 484
--- End of stack trace from previous location ---
at DotNet.Testcontainers.Configurations.WaitStrategy.<>c__DisplayClass1_0.<g__UntilAsync|0>d.MoveNext() in //src/Testcontainers/Configurations/WaitStrategies/WaitStrategy.cs:line 72
--- End of stack trace from previous location ---
at DotNet.Testcontainers.Configurations.WaitStrategy.WaitUntilAsync(Func`1 wait, TimeSpan frequency, TimeSpan timeout, CancellationToken ct) in //src/Testcontainers/Configurations/WaitStrategies/WaitStrategy.cs:line 98
at DotNet.Testcontainers.Containers.DockerContainer.UnsafeStartAsync(CancellationToken ct) in //src/Testcontainers/Containers/DockerContainer.cs:line 506
at DotNet.Testcontainers.Containers.DockerContainer.StartAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 291
Error Message:
Docker.DotNet.DockerApiException : Docker API responded with status code=Conflict, response={"message":"Container 2d2ceb17cfcc78d635c0ae00c2ae250a60912715323deefeaaab8e7faf4fab80 is not running"}

The error message indicates that the container is deleted before the test runs. Additionally, I sometimes get a different error message even when running the same code:

[testcontainers.org 00:00:00.05] Connected to Docker:
Host: npipe://./pipe/docker_engine
Server Version: 24.0.7
Kernel Version: 10.0 20348 (20348.1.amd64fre.fe_release.210507-1500)
API Version: 1.43
Operating System: Microsoft Windows Server Version 21H2 (OS Build 20348.2461)
Total Memory: 64.00 GB
[testcontainers.org 00:00:00.07] Searching Docker registry credential in Auths
[testcontainers.org 00:00:00.07] Searching Docker registry credential in CredsStore
[testcontainers.org 00:00:00.07] Searching Docker registry credential in Auths
[testcontainers.org 00:00:00.07] Searching Docker registry credential in CredHelpers
[testcontainers.org 00:00:00.08] Docker registry credential mcr.microsoft.com not found
[testcontainers.org 00:06:39.42] Docker image mcr.microsoft.com/cosmosdb/windows/azure-cosmos-emulator:latest created
[testcontainers.org 00:06:39.68] Docker container 2200453bdcf9 created
[testcontainers.org 00:06:39.71] Start Docker container 2200453bdcf9
[testcontainers.org 00:07:01.74] Wait for Docker container 2200453bdcf9 to complete readiness checks
[testcontainers.org 00:07:01.74] Execute "PowerShell -Command Exit(-Not((Test-NetConnection -ComputerName 'localhost' -Port 8081).TcpTestSucceeded))" at Docker container 2200453bdcf9
[testcontainers.org 00:07:18.00] Execute "PowerShell -Command Exit(-Not((Test-NetConnection -ComputerName 'localhost' -Port 8081).TcpTestSucceeded))" at Docker container 2200453bdcf9
[testcontainers.org 00:07:28.79] Execute "PowerShell -Command Exit(-Not((Test-NetConnection -ComputerName 'localhost' -Port 8081).TcpTestSucceeded))" at Docker container 2200453bdcf9
[testcontainers.org 00:07:38.75] Docker container 2200453bdcf9 ready
[xUnit.net 00:08:15.75] Test.Integration.TestSomething [FAIL]
Failed Test.Integration.TestSomething [1 ms]
Error Message:
Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: ServiceUnavailable (503); Substatus: 20003; ActivityId: 00000000-0000-0000-0000-000000000000; Reason: (GatewayStoreClient Request Timeout. Start Time UTC:5/21/2024 12:57:16 AM; Total Duration:36016.7705 Ms; Request Timeout 20000 Ms; Http Client Timeout:65000 Ms; Activity id: 00000000-0000-0000-0000-000000000000;);; Diagnostics:{"Summary":{},"name":"Account Read","start datetime":"2024-05-21T00:57:16.348Z","duration in milliseconds":36021.615}
---- System.Threading.Tasks.TaskCanceledException : The operation was canceled.
-------- System.Threading.Tasks.TaskCanceledException : The operation was canceled.
------------ System.IO.IOException : Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
---------------- System.Net.Sockets.SocketException : The I/O operation has been aborted because of either a thread exit or an application request.

It seems like the container readiness checks are failing intermittently, causing the tests to fail.

Did I miss something in my configuration? Any advice on how to resolve these issues would be greatly appreciated.

[kiview]

Thanks for sharing this @noh626.

There is an upstream issue with the image and GHA (see Azure/azure-cosmos-db-emulator-docker#56), and I guess AzurePipelines will use similar runners in some way, so I imagine it can be related.

[noh626]

Thanks, @HofmeisterAn can you suggest a more reliable wait strategy?

[HofmeisterAn]

As mentioned, I do not believe this issue is related to the wait strategy. The container is crashing. You will need to look into the container logs (of the affected container). Typically, the logs contain information about the crash.

[noh626]

Thanks @HofmeisterAn,
I am encountering an issue while trying to retrieve logs from a container using the GetLogsAsync method. I got the following error:

Message: 
System.IO.IOException : unknown stream type

  Stack Trace: 
MultiplexedStream.ReadOutputAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
MultiplexedStream.CopyOutputToAsync(Stream stdin, Stream stdout, Stream stderr, CancellationToken cancellationToken)
MultiplexedStream.ReadOutputToEndAsync(CancellationToken cancellationToken)
DockerContainerOperations.GetLogsAsync(String id, TimeSpan since, TimeSpan until, Boolean timestampsEnabled, CancellationToken ct) line 77
CosmosDbEmulatorFixture.InitializeAsync() line 73

Additionally, when I use the WithLogger method, it only outputs the image pulling progress and does not provide any useful information for the container startup process.

Could you please advise on how I can obtain the detailed logs from the container?

[noh626]

I managed to capture the logs for the failing test cases as shown below:

"C:\Windows\system32\cmd.exe" /D /E:ON /V:OFF /S /C "CALL "C:\_work\_temp\5126b252-72c4-4f51-b800-9e1101952fb9.cmd""
Transcript started, output file is C:\CosmosDB.Emulator\bind-mount\Diagnostics\Transcript.log
Wait-CosmosDbEmulator : The emulator failed to reach Running status within 300 seconds
At C:\Program Files\Azure Cosmos DB Emulator\PSModules\Microsoft.Azure.CosmosDB.Emulator\Microsoft.Azure.CosmosDB.Emulator.psm1:654 char:12
+     [void](Wait-CosmosDbEmulator -AlternativeInstallLocation $Emulato ...
+            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Wait-CosmosDbEmulator
 

Have you seen this issue before? Any insights on how to troubleshoot this further would be appreciated.

[HofmeisterAn]

Have you seen this issue before?

No, I haven't — sorry. The Microsoft.Azure.CosmosDB.Emulator.psm1 PowerShell Module does not belong to Testcontainers. Maybe the developers at Microsoft can help here. The only (similar) issue I found in their repository is: Azure/azure-cosmos-db-emulator-docker#38.

Maybe you can copy the PowerShell Module from the container to your host to get a better understand what exactly fails (sources of line 654).