[Enhancement]: Being able to use Ryuk in (CI) environment where Docker "userns-remap" mode is active

[dalbani]

Module

Core

Proposal

Hello,

I'm using Testcontainers in an on-prem GitLab CI environment, where the GitLab runner hosts have their Docker server with userns-remap turned on.
I ran into a couple of issues related to Ryuk in this case.
First, from what I read on https://java.testcontainers.org/features/configuration/#customizing-ryuk-resource-reaper, the Ryuk container must be started as a privileged container.
That's the reason why I get the following error when Testcontainers tried launching the Ryuk container:

...
com.github.dockerjava.api.exception.BadRequestException: Status 400: {"message":"privileged mode is incompatible with user namespaces.  You must run the container in the host namespace when running privileged mode"}
...

As a quick hack, I worked around this issue by "monkeypatching" org.testcontainers.utility.RyukContainer and adding:

    cmd
        .getHostConfig()
        ...
        .withUsernsMode("host")  /// <= added

The Ruyk container could then start fine.

But due to the fact the GitLab runner container conforms to this userns-remap mode (and thus not Ruyk as per the above), Testcontainers was not able to connect to Ryuk:

...
WARN --- [containers-ryuk] o.t.utility.RyukResourceReaper : Can not connect to Ryuk at 172.17.0.1:35592

Again, I had to resort to monkeypatching org.testcontainers.utility.RuykResourceReaper to have:

...
// String host = ryukContainer.getHost();
String host = ryukContainer.getCurrentContainerInfo().getNetworkSettings().getIpAddress();
// Integer ryukPort = ryukContainer.getFirstMappedPort();
Integer ryukPort = ryukContainer.getExposedPorts().get(0);
...

My understanding is that this fix is obviously not portable in environments where the Docker host is not local to where Testcontainers is running.
But bar changing my CI environment setup to drop the userns-remap mode, I don't see any other solution to my issue.

What do you think of having "official support" for such a workaround in Testcontainers, so I don't have to monkeypatch Java classes... 🤓
If not via a setting (e.g. ryuk.container.userns_mode=host or something related), maybe via a way to customize the behavior of Testcontainers dynamically?

Thanks!

[eddumelendez]

Hi, have you tried disabling privileged mode? Testcontainers documentation tells about how to customize docker host detection.

[dalbani]

Hi, have you tried disabling privileged mode?

Doh, I should have tried that first.
And, yes indeed, I've just tested and it seems to work.
I removed my monkeypatch of RuykContainer, set ryuk.container.privileged=false and my integration test still run / pass.
Thanks for the suggestion! 👍
Although I don't know what to think of the assertion on https://java.testcontainers.org/features/configuration/#customizing-ryuk-resource-reaper: "Ryuk must be started as a privileged container."

And now regarding RuykResourceReaper.

Testcontainers documentation tells about how to customize docker host detection.

Good suggestion, it just occurred to me that Docker host customization what not only about the hostname 🙄
Although I guess that changing the Docker host will affect more things than only what happens in RuykResourceReaper?
Anyway, I'll try it out and see 😄

[dalbani]

Anyway, I'll try it out and see 😄

Actually, unless I'm missing something in my understanding of Testcontainers, I don't think tweaking the Docker host detection mechanism is an option here.
As overriding the Docker host globally doesn't seem to make sense when the goal is only for RyukResourceReaper to be able to talk to Ryuk.
Does it?

For the record, my GitLab CI mounts the Docker socket inside the GitLab runner container.
So I suppose that it is the code in DockerClientProviderStrategy that is applicable to my case:

            case "unix":
            case "npipe":
                if (DockerClientConfigUtils.IN_A_CONTAINER) {
                    return client
                        .inspectNetworkCmd()
                        .withNetworkId("bridge")
                        .exec()
                        .getIpam()
                        .getConfig()
                        .stream()
                        .filter(it -> it.getGateway() != null)
                        .findAny()
                        .map(Network.Ipam.Config::getGateway)
                        .orElseGet(() -> {
                            return DockerClientConfigUtils.getDefaultGateway().orElse("localhost");
                        });
                }

[eddumelendez]

You need to set TESTCONTAINERS_HOST_OVERRIDE

[dalbani]

I don't think TESTCONTAINERS_HOST_OVERRIDE will help me here.
For the record, here are the network details of the Ryuk container:

        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "5215646a4949ec04ff942c04e178f5822be7c4beb3d728716de5ce930255e11a",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "8080/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "35597"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "35597"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/5215646a4949",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "45b34656101e1c13b33fbdc3b46c3611e766444c28dbb754a882a5070970e11b",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "2001:67c:2e8:102:a66:3f:0:3",
            "GlobalIPv6PrefixLen": 96,
            "IPAddress": "172.17.0.3",
            "IPPrefixLen": 16,
            "IPv6Gateway": "2001:67c:2e8:102:a66:3f:0:1",
            "MacAddress": "02:42:ac:11:00:03",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "fc11a9fb9846a8dc9e4cf8060b8ebfbd2e7f18a0dba41a235d02e19028e85ea4",
                    "EndpointID": "45b34656101e1c13b33fbdc3b46c3611e766444c28dbb754a882a5070970e11b",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.3",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "2001:67c:2e8:102:a66:3f:0:1",
                    "GlobalIPv6Address": "2001:67c:2e8:102:a66:3f:0:3",
                    "GlobalIPv6PrefixLen": 96,
                    "MacAddress": "02:42:ac:11:00:03",
                    "DriverOpts": null
                }
            }
        }

So instead of connecting to 172.17.0.1:35597 (which is not reachable from the GitLab runner container due to userns-remap), I need to have Testcontainers connect to 172.17.0.3:8080.
Setting TESTCONTAINERS_HOST_OVERRIDE to say, 172.17.0.3, would still amount to Testcontainers trying to connect to 172.17.0.3:35597.

[eddumelendez]

I need to have Testcontainers connect to 172.17.0.3:8080

Do you need the fixed port? Testcontainers works by providing random ports and avoid port collisions, which enables parallelism.

I am not sure about the infrastructure but you need the VM IP. See some examples of how to set TESTCONTAINERS_HOST_OVERRIDE for other container runtimes.

Worst case scenario would be to disable ryuk.

[dalbani]

In my particular case, 10.102.0.63 is the IP address of the VM.
But the Docker server only binds exposed ports to 172.17.0.1, as expected.
So even with TESTCONTAINERS_HOST_OVERRIDE set to 10.102.0.63, Testcontainers fails at connecting to 10.102.0.63: 35597 (the random port in question in this particular example).

Yes, disabling Ryuk would "solve" my problem, but I'm precisely trying to use Ryuk, to ensure that my CI environment doesn't accumulate leftovers in case jobs fail.
As I exposed above, I did get it to work by applying small changes to the RyukContainer and RuykResourceReaper classes.
But actual monkeypatching is not really an option in Java, as I had to override the classes in the classpath...
Which is obviously not maintainable.

[eddumelendez]

the change about RyukContainer can be take it into account but the one about RyukResourceReaper not sure for this use case.

[kiview]

Hey @dalbani , you said:

So instead of connecting to 172.17.0.1:35597 (which is not reachable from the GitLab runner container due to userns-remap), I need to have Testcontainers connect to 172.17.0.3:8080.

If your GitLab Runner can't reach the gateway interface that exposed the dynamically mapped port, how would your tests be able to interact with others containers, that are not Ryuk? Meaning, even if we solve this issue with Ryuk, I don't understand how it would solve your use case.

[dalbani]

Hi @kiview, thanks for chipping in.

The reason I can get away with not being able to access exposed ports on the Docker host, is that my test uses Docker Compose [*].
Which creates a new, isolated network which my test, e.g. a Playwright browser, can connect to without a problem.
Well, actually, I need to use a trick: I have to dynamically attached the GitLab runner container to the Docker Compose network (docker network connect [network-id] [runner-container-id].

So, the only stumbling block I have is the ability to tweak some Ryuk related settings, as I did in RyukContainer and RyukResourceReaper as exposed above.
I'm very much aware that my need is quite specific — although am I really the only one who wants to be able Ryuk in a CI environment with userns-remap? — so maybe we could introduce some generic "callbacks" for the classes in question?
In order for users to be able to dynamically tweak some settings.

[*] I'm aware that it's not the solution you personally recommend, based on a previous message of yours I saw somewhere in another discussion.
But the reason why I prefer to rely on Docker Compose, is that I can use the Docker Compose definition for local development as well.