[Question] Use existing container if it exists

[lonix1]

I'm using this library to setup containers for integration tests for postgres. It works very well.

However it is slow to start and stop, as to be expected. So what I'm trying to achieve:

• On my dev machine, I have another instance of postgres running (almost) all the time. If it is running then I'd like the library to use it.
• On the test server, no containers would be running, so I'd like the library to start/stop throwaway containers as usual.

Is this possible somehow?

(I could add boilerplate code to detect if the container is running, etc,, but I'm hoping this library has functionality to do that?)

[HofmeisterAn]

Right now, .NET Testcontainers does not support container reuse. Although, some Docker resources like network and volume can be reused. I think reusing a container does not match the concept. Starting and stopping should not take that much time (with cached images). There are probably other / better ways to optimize the performance (development experience).

[lonix1]

Understood, thanks.

For anyone who also needs this, what I did was:

• use Docker.DotNet library to determine if container is running (1)
• if not, create new container using this library (2)
• grab connection string from (1) or (2)

[PSanetra]

@HofmeisterAn I am currently also thinking about the container reuse concept. We have some containers that take a while to start (not just pulling the image, but the starting procedure itself). I have measured about 30 seconds in our case. This makes the local development and test feedback cycle very slow. Reusing containers could really improve this.

I have seen that such a reuse feature is already supported for Java Testcontainers, although it is not documented: testcontainers/testcontainers-java#1781

Do you think working on such a contribution for .NET Testcontainers makes sense?

[HofmeisterAn]

I still think reusing containers (resources) does not fit very well into the concept of throwaway instances, especially on CI pipelines. But still, keeping containers is useful for development. For example, I use multiple test sessions (one that starts the container, others to execute the tests), but that requires manual work too.

I'm just concerned that features, like this, will mess up with the tests and develop will use resources that do not belong to their tests. I haven't looked much into the Java implementation, but it looks like they're using a much better approach to identify resources. My first idea, was just the container name.

All we need to do is, to set the container field in TestcontainersContainer to the identified resource (we need to think about other resources too):

https://github.com/HofmeisterAn/dotnet-testcontainers/blob/de4c4a251152ee2cfdcd4061c8b1bcc0621e692f/src/DotNet.Testcontainers/Containers/TestcontainersContainer.cs#L165

Do you think working on such a contribution for .NET Testcontainers makes sense?

Of course, contribution is always welcome.

[swissarmykirpan]

@PSanetra could the tests perhaps be run in parallel, so that the 30 second cost is only felt once?

[PSanetra]

@swissarmykirpan I think the possibility to run the tests in parallel if the test containers are reused, will depend on the container. I guess in many cases it should be possible to run tests in parallel even if the containers are reused, but it is necessary to look at specific cases.

[swissarmykirpan]

@PSanetra sorry I don't think I made myself understood correctly. What I meant to say was that if test execution is serial (one at a time), then yes every test spins up a container, once the previous test has completed. However, if the test execution is parallel, then all the containers of the parallel tests will be started at the same time, and so the impact of having to wait 30 seconds multiple times goes away, without having to reuse containers.

Please tell me if I am making any sense here

[HofmeisterAn]

A good approach will be to generate a hash of the builder configuration and assign it as a label to the Docker resource. If we find the same hash, we can reuse the resource.

[iikuzmychov]

@swissarmykirpan You are right that we can use a single container to run a set of tests and it will be work fast. But during test development, we need to run a single test method many times which causes creating a new container and spend time. This is why we want smth like .WithReuse(true)

[benjaminsampica]

+1 on this. I use TestContainers for local development as well as testing because it's just too darn easy. This feature would prevent me from doing something I'm having to do custom today!

[HofmeisterAn]

I will look into generating hash values for the builder configurations and think about an MVP. I think if we can reliably label resources with an identifier, then there is not much work left.

[Gavamot]

I am really need .WithReuse(true). Why it exists for Java but for .NET doesn't?

[Jump33]

This is a very important feature! please implement it for DOTNET

[HofmeisterAn]

I am really need .WithReuse(true). Why it exists for Java but for .NET doesn't?

Because no one has implemented it so far. Would you like to do it? Then I can spend my spare time doing something else 🏖️.

[bsideup]

FYI we are looking into adding such feature to Testcontainers Desktop so that any Testcontainers version would have this capability, implemented consistently across the languages. Stay tuned!

[HofmeisterAn]

In case someone else would like to take a look at it until I find some time, I think implementing the reusing feature is fairly simple. We just need to assign this field to the existing container inspect response (this works exactly the same for the other resources). The only question is how to identify them reliably. A very simple approach could use a fixed label at the beginning as well.

[david-szabo97]

Implemented this in PR #1051
It's not a perfect implementation, but hopefully, it will be enough to get others to chime in and perfect it together. 🚀

[glen-84]

I wonder if all the complexity of creating a hash from the builder configuration, ignoring certain properties, etc. is necessary?

Would it not work to simply have an API similar to:

WithReuse(key: "my-unique-key")

... and store that key as a label?

Or is it important for this to work without having to specify a "key"?

[HofmeisterAn]

Thanks to @david-szabo97, we have, in my opinion, a very good initial implementation that supports reusing resources 🙏. There are a few minor things left and some limitations, which I plan to address/document in the next few days. However, I'd appreciate early feedback from you: #1051.

If your development workflow could benefit from the reuse feature and if you have some time available, please take a look at the PR and consider experimenting with the feature. Please note that the reuse feature is not meant to replace a proper singleton implementation running in a single process (it should not be used in the manner the test demonstrates). To enable reuse and prevent resources from being cleaned up, use the following builder configuration:

_ = new ContainerBuilder()
    .WithReuse(true)
    .WithCleanUp(false)

In the future, only WithReuse(bool) will be necessary. We will be adding additional checks as well. Please note that not all builder APIs are currently considered when generating the hash value to detect an existing resource yet.

[cbrevik]

Is there a way for me to test this experimental feature directly in my solution? I've been looking around for artifacts in the PR checks for built .nupkg which I can depend on locally. But maybe it doesn't do that in PR runs?

I am really excited about this feature though, I think it would save us a lot of time running all of our E2E-tests continually when developing locally. Also we are considering using TestContainers for local development so we could quickly bootstrap local instances of for example databases, and here "re-use" would cut down a lot of the lead-time when re-building/running our solution.

[HofmeisterAn]

No, there is no snapshot or beta version that includes the experimental feature. You need to clone the repository to test it. Sorry.

[MichaelLogutov]

@HofmeisterAn I've tried and currently have errors after using .WithReuse(true).WithCleanUp(false). Does this feature supports custom network (I'm using bridge)? And should I stop disposing containers for this feature to work properly?

[HofmeisterAn]

What kind of issue are you experiencing?

[MichaelLogutov]

Here stripped down repo. This code works only the first time. Maybe this is because on reuse random assigned ports are not restored?

using System.Text;
using Confluent.Kafka;
using DotNet.Testcontainers.Builders;
using DotNet.Testcontainers.Configurations;
using DotNet.Testcontainers.Containers;

namespace Sandbox;

static class Program
{
    private const string KafkaHostname = "kafka";
    private const ushort KafkaPort = 9092;
    
    static async Task Main()
    {
        await using var network = new NetworkBuilder()
            .WithDriver(NetworkDriver.Bridge)
            .WithReuse(true)
            .WithCleanUp(false)
            .Build();
        
        await using var container = new ContainerBuilder()
            .WithImage("confluentinc/confluent-local:7.5.3")
            .WithHostname(KafkaHostname)
            .WithNetwork(network)
            .WithNetworkAliases(KafkaHostname)
            .WithPortBinding(KafkaPort, assignRandomHostPort: true)
            .WithEnvironment("KAFKA_LISTENERS", $"PLAINTEXT://{KafkaHostname}:29092,CONTROLLER://{KafkaHostname}:29093,PLAINTEXT_HOST://0.0.0.0:{KafkaPort}")
            .WithEnvironment("KAFKA_CONTROLLER_QUORUM_VOTERS", $"1@{KafkaHostname}:29093")
            .WithStartupScript(c =>
                $"""
                 #!/bin/bash
                 export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://{c.IpAddress}:29092,PLAINTEXT_HOST://{c.Hostname}:{c.GetMappedPublicPort(KafkaPort)}
                 /etc/confluent/docker/run
                 """)
            .WithWaitStrategy(Wait.ForUnixContainer().UntilMessageIsLogged("Server started"))
            .WithReuse(true)
            .WithCleanUp(false)
            .Build();

        await container.StartAsync();
        
        var producer_config = new ProducerConfig
        {
            BootstrapServers = $"{container.Hostname}:{container.GetMappedPublicPort(KafkaPort)}"
        };

        Console.WriteLine(string.Join(";", producer_config.Select(kv => $"{kv.Key}={kv.Value}")));

        using var producer = new ProducerBuilder<Null, string>(producer_config).Build();
        await producer.ProduceAsync("test", new Message<Null, string> { Value = "Hello, world!" });
        
        Console.WriteLine("Message successfully published");
    }
    
    public static ContainerBuilder WithStartupScript(this ContainerBuilder builder, Func<IContainer, string> script)
    {
        const string startup_script_file_path = "/testcontainers.sh";

        return builder
            .WithEntrypoint("/bin/sh", "-c")
            .WithCommand("while [ ! -f " + startup_script_file_path + " ]; do sleep 0.1; done; " + startup_script_file_path)
            .WithStartupCallback((container, ct) =>
            {
                var script_content = script(container).Replace("\r", "");
                return container.CopyAsync(Encoding.Default.GetBytes(script_content), startup_script_file_path, Unix.FileMode755, ct);
            });
    }
}

[HofmeisterAn]

I took a quick look and gave your configuration a try. TBH, I was pretty surprised. Considering you are attempting to reuse Kafka (which is not simple to set up), it actually works very well and kind of as expected.

Does this feature supports custom network (I'm using bridge)?

I guess you are referring to the log message that the network will be recreated since it could not find a reusable hash. This is because you did not assign a network name. By default, the network builder will assign a random name (which changes the hash). Use WithName(string) and TC will reuse the network too.

Coming back to the Kafka issue, the initial create and start will configure Kafka (advertised.listeners) using a randomly assigned host port. When you rerun your tests, a new random host port will be assigned. This is because we stop the container after the test run. This issue is a bit special and may require adjustments in how you/we configure Kafka. Most other modules should not be affected by this issue.

To test the actual reuse, you can remove the using declaration from the container (temporary solution). Then your configuration will work.

Maybe this is because on reuse random assigned ports are not restored?

As mentioned above, they are, but they change and are not the same as Kafka was initially configured.

[MichaelLogutov]

That makes sense.
Are there a way to execute some code inside container after it's been reused (started the second time)?

To test the actual reuse, you can remove the using declaration from the container (temporary solution). Then your configuration will work.

I thought that WithCleanUp(false) disables dispose just as well?

[HofmeisterAn]

I thought that WithCleanUp(false) disables dispose just as well?

It prevents the resource from being cleaned up. The container will still be stopped, though. Removing the using declaration will keep the container running; therefore, it won't allocate a new random host port.

Are there a way to execute some code inside container after it's been reused (started the second time)?

Hmm, there is nothing built into the reuse feature yet. But starting the container will execute the startup callback again (something that could be utilized together with the exec API, I think).

[MichaelLogutov]

Yeah, removing dispose call does the trick. Using your branch with .WithReuse(true).WithCleanUp(false) under #if DEBUG alongside with hardcoded names and disabled dispose helped - I can debug tests a lot faster (I've got like 5-8 containers to run for every test).

[Barsonax]

I noticed that when you run multiple test assemblies in parallel (which is currently how dotnet test works by default) which all create the same reusable container multiple containers are actually created. I suspect some concurrency issue.

Is this a known issue and will this be fixed or do we have to prevent these situations ourselves?

[InspiringCode]

Could you possibly provide a prerelease on NuGet, so we can already try use this in our project? This would be very helpful.

[0xced]

Reuse support was released in Testcontainers 3.8.0.

I think this issue can be closed.

[0xced]

@Barsonax I'd suggest opening a new issue for the possible concurrency issue.

[HofmeisterAn]

run multiple test assemblies in parallel

This is basically out of scope. It is a development feature. We are not synchronizing between processes or between parallel running tests in the same assembly (e.g. xUnit.net's test collections). You need to take care of it right now. As mentioned in the docs, this feature does not replace proper singleton implementations, too.

As @0xced mentioned, the basic feature is implemented and shipped in version 3.8.0. I think closing the issue and discussing further enhancements in follow-up issues is a good idea 🙌.

Thanks for all the contributions and feedback.