IContainer.State property is not updated after container dies

[jorik]

Hello,

I have a question regarding the IContainer.State property. I'd like to be notified when my container dies. I would expect the IContainer.State property to hold real-time state information, but it does not seem to do so.

Then I tried to use the IContainer.Stopped event, but that also does not seem to update.

What is the best way to be notified when a container dies?

Here is my test code. It starts a container that dies after 10 seconds.

using System.Diagnostics;
using Docker.DotNet.Models;
using DotNet.Testcontainers.Builders;
using DotNet.Testcontainers.Containers;
using Xunit;
using Xunit.Sdk;

namespace testcontainers_state_update;

public class UnitTest1
{
    [Fact]
    public async Task State_Does_Not_Get_Updates()
    {
        var container = new ContainerBuilder()
            .WithImage("ubuntu:22.04")
            .WithCommand("sleep", "10")
            .Build();

        container.Started += (sender, args) => Console.WriteLine("Started");
        container.Stopped += (sender, args) => Console.WriteLine("Stopped");

        await container.StartAsync();

        var maxWait = TimeSpan.FromSeconds(20);
        var stopwatch = Stopwatch.StartNew();
        while (stopwatch.Elapsed < maxWait)
        {
            if (container.State == TestcontainersStates.Exited)
                return;

            await Task.Delay(100);
        }

        throw new XunitException("Container state is not updated");
    }
}

Here are the logs. Note only Started has been logged, and not Stopped.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[testcontainers.org 00:00:00.04] Connected to Docker:
  Host: unix:///Users/anonymous/.docker/run/docker.sock
  Server Version: 24.0.2
  Kernel Version: 5.15.49-linuxkit-pr
  API Version: 1.43
  Operating System: Docker Desktop
  Total Memory: 7.76 GB
[testcontainers.org 00:00:00.11] Docker container 30ff66faef5b created
[testcontainers.org 00:00:00.13] Start Docker container 30ff66faef5b
[testcontainers.org 00:00:00.28] Wait for Docker container 30ff66faef5b to complete readiness checks
[testcontainers.org 00:00:00.28] Docker container 30ff66faef5b ready
[testcontainers.org 00:00:00.30] Docker container eb7d8136798e created
[testcontainers.org 00:00:00.31] Start Docker container eb7d8136798e
[testcontainers.org 00:00:00.43] Wait for Docker container eb7d8136798e to complete readiness checks
[testcontainers.org 00:00:00.44] Docker container eb7d8136798e ready
Started
[xUnit.net 00:00:20.76]     testcontainers_state_update.UnitTest1.State_Does_Not_Get_Updates [FAIL]
  Failed testcontainers_state_update.UnitTest1.State_Does_Not_Get_Updates [20 s]
  Error Message:
   Container state is not updated
  Stack Trace:
     at testcontainers_state_update.UnitTest1.State_Does_Not_Get_Updates() in /Users/anonymous/projects/test/testcontainers-state-update/UnitTest1.cs:line 35
--- End of stack trace from previous location ---

Failed!  - Failed:     1, Passed:     0, Skipped:     0, Total:     1, Duration: < 1 ms - testcontainers-state-update.dll (net8.0)

[HofmeisterAn]

I am sorry, but we do not support this yet. We do not receive notifications from the Docker API when a container terminates. IIRC, Docker.DotNet does offer support for monitoring events. Perhaps we could leverage this to detect when a container terminates and then forward this information through an event. WDYT?

[jorik]

Hey @HofmeisterAn, thanks for answering so quickly :D

That does sound like a proper solution indeed!

Is the state currently only updated if the container is stopped via the IContainer interface itself?

As a workaround I'm currently doing the following to detect if the container has stopped. Do you see any issues with this?

try
{
    // Try to execute a command in the container. If it fails, the container is not running anymore
    await container.ExecAsync(new[] {"ls"});
}
catch (DockerApiException e) when (e.StatusCode is HttpStatusCode.Conflict or HttpStatusCode.NotFound)
{
    // Container is not running anymore
}

[HofmeisterAn]

Is the state currently only updated if the container is stopped via the IContainer interface itself?

Yes, the field is updated by calling StopAsync():

testcontainers-dotnet/src/Testcontainers/Containers/DockerContainer.cs

Lines 475 to 483 in 69292d1

	try
	{
	_container = await _client.InspectContainerAsync(_container.ID, ct)
	.ConfigureAwait(false);
	}
	catch (DockerApiException)
	{
	_container = new ContainerInspectResponse();
	}

As a workaround I'm currently doing the following to detect if the container has stopped. Do you see any issues with this?

I am not sure what you are trying to achieve, but having a terminating container is unusual and could likely be a misconfiguration, except for one-shot configurations. Of course, you can use it to detect a terminated container, but having it integrated into TC would be a better approach.

[jorik]

The terminating container is actually something that I want to explicitly test :) I'm building a test suite around a server agent, and in some cases the agent should terminate. In some situations, the only way to verify that it successfully shut down is by checking the container status (== Exited).

I'm using Testcontainers to run the agent in my tests, with different docker images (ubuntu 18, 20, 22, etc)

[HofmeisterAn]

Okay, in that case, you may want to consider implementing a custom wait strategy (I was already wondering because your example did not specify one). Probably something like this will work, and then you should be able to get the container's state.

testcontainers-dotnet/examples/Flyway/tests/Flyway.Tests/DbFixture.cs

Lines 63 to 73 in 69292d1

	private sealed class MigrationCompleted : IWaitUntil
	{
	// The Flyway container will exit after executing the database migration. We do not
	// check if the migration was successful. To verify its success, we can either
	// check the exit code of the container or the console output, respectively the
	// standard output (stdout) or error output (stderr).
	public Task<bool> UntilAsync(IContainer container)
	{
	return Task.FromResult(TestcontainersStates.Exited.Equals(container.State));
	}
	}

If you want to abort the startup process after a timeout, you can pass a cancellation token to StartAsync().