New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot Use testcontainers with Isolated Containers #712

Closed
schrieveslaach opened this Issue May 29, 2018 · 12 comments

Comments

Projects
None yet
6 participants
@schrieveslaach

schrieveslaach commented May 29, 2018

I do have a docker setup, providing container isolation with a user namespace. Following configuration:

/etc/docker/daemon.json

{
    "userns-remap": "default"
}

/etc/subuid

dockremap:1476256:65536

/etc/subgid

dockremap:1476256:65536

When I use mvn test which tries to start containers, I get following error (cannot connect to Ryuk):

[main] INFO org.testcontainers.DockerClientFactory - Docker host IP address is localhost
[main] INFO org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 18.05.0-ce
  API Version: 1.37
  Operating System: Arch Linux
  Total Memory: 3948 MB
[testcontainers-ryuk] WARN org.testcontainers.utility.ResourceReaper - Can not connect to Ryuk at localhost:32770
java.net.ConnectException: Verbindungsaufbau abgelehnt (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at java.net.Socket.connect(Socket.java:538)
        at java.net.Socket.<init>(Socket.java:434)
        at java.net.Socket.<init>(Socket.java:211)
        at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:119)
        at java.lang.Thread.run(Thread.java:748)
[testcontainers-ryuk] WARN org.testcontainers.utility.ResourceReaper - Can not connect to Ryuk at localhost:32770
java.net.ConnectException: Verbindungsaufbau abgelehnt (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at java.net.Socket.connect(Socket.java:538)
        at java.net.Socket.<init>(Socket.java:434)
        at java.net.Socket.<init>(Socket.java:211)
        at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:119)
        at java.lang.Thread.run(Thread.java:748)

I assume that the ryuk container must be started with the command line option --userns=host (docker ignores user namespaces).

Is it possible to integrate this commandline option?

@kiview

This comment has been minimized.

Member

kiview commented Jun 12, 2018

AFAIK docker-java doesn't suport user namespaces ATM.

@nathanburrell

This comment has been minimized.

nathanburrell commented Sep 28, 2018

Rather than disabling userns remapping on the ryuk container, we can investigate why the ryuk container cant run with that feature enabled as that would be the better fix.

As some environments may never ever allow a user to turn of the remapping feature for a container. As some cloud CI/CD providers (such as bitbucket pipelines and others) run in a multi tenanted environment and have user namespace remapping always enabled for security purposes.

@bsideup

This comment has been minimized.

Member

bsideup commented Sep 28, 2018

@nathanburrell is it possible to turn it on locally on Mac? Then I can experiment with it locally and maybe figure out why it doesn't work :)

@nathanburrell

This comment has been minimized.

nathanburrell commented Sep 28, 2018

It sure is, you can specify daemon args following the steps here

https://docs.docker.com/v17.12/docker-for-mac/#daemon-tab

The Arg you want to provide is
--userns-remap=default

More information about user namespace remapping is here https://docs.docker.com/engine/security/userns-remap/

@nathanburrell

This comment has been minimized.

nathanburrell commented Oct 9, 2018

Any updates on this?

Over the coming months we will be removing the exclusion list as it leaves potential security holes within our product, and I want to ensure test containers still works with pipelines when that time comes.

@kiview

This comment has been minimized.

Member

kiview commented Oct 9, 2018

I'll try to see if I can reproduce it locally on Linux when setting the flag on the daemon.
The actual ryuk implementation is pretty slim, so I wonder where the problem comes from:
https://github.com/testcontainers/moby-ryuk/blob/master/main.go

@schrieveslaach

This comment has been minimized.

schrieveslaach commented Oct 10, 2018

@kiview, if you have a new SNAPSHOT version, I can test it on my machine.

@rnorth

This comment has been minimized.

Member

rnorth commented Dec 13, 2018

I was trying to reproduce the other day so that we can look into why - however, I had trouble with the userns-remap setting on Docker for Mac. It seems that setting this in the daemon JSON configuration causes the daemon to fail to start up.

I haven't yet had time to try this in another context (e.g. with docker-machine).

Sorry for there being no progress made on this ticket so far. If anybody has ability to diagnose why this setting is incompatible with Ryuk, we'd be keen to get your input.

@jrehwaldt

This comment has been minimized.

jrehwaldt commented Dec 14, 2018

Suppose the issue with bitbucket pipeline is indeed the same you can test it against their dockerized pipeline runner as follows:

docker run --privileged --name bitbucket-docker -d -p 127.0.0.1:12375:2375 --memory=1g --memory-swap=1g --memory-swappiness=0  atlassian/pipelines-docker-daemon:prod-stable
set DOCKER_HOST=tcp://127.0.0.1:12375
// run testcontainers mounting against DOCKER_HOST

You should see the same log as posted in the issue description.

@rnorth

This comment has been minimized.

Member

rnorth commented Dec 15, 2018

Ah, that's a good suggestion, thanks @jrehwaldt. I'll give this a try and see what I can find.

@rnorth

This comment has been minimized.

Member

rnorth commented Dec 15, 2018

Hmm, either the userns-remapping or the docker-in-docker nature of this is meaning that none of the exposed ports on any containers are accessible from Mac (not just Ryuk). I imagine this might be a limitation of Docker for Mac's TCP proxy.

I think we'll need to do this on Linux, though, so I'll pursue that avenue.

@rnorth

This comment has been minimized.

Member

rnorth commented Dec 15, 2018

We seem to have two issues for the same thing (this and #700), and it will get confusing having both. As such, I'll close this as the newer of the two. Please continue discussion as needed on #700.

@rnorth rnorth closed this Dec 15, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment