New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not working in Bitbucket pipeline: "authorization denied by plugin pipelines: Command not supported." #700

Closed
4ntoine opened this Issue May 17, 2018 · 41 comments

Comments

Projects
None yet
@4ntoine
Copy link

4ntoine commented May 17, 2018

It seems to be both docker and testcontainers issue:

Running name.antonsmirnov.apptogether.dao.DeveloperDaoTest
08:08:56.040 [main] INFO  EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
08:08:56.060 [main] INFO  DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=tcp://localhost:2375
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='root'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=tcp://localhost:2375,registryUsername=root,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/root/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

08:08:56.066 [main] INFO  DockerClientFactory - Docker host IP address is localhost
08:08:56.227 [main] INFO  DockerClientFactory - Connected to docker: 
  Server Version: 17.12.0-ce
  API Version: 1.35
  Operating System: Alpine Linux v3.7 (containerized)
  Total Memory: 32171 MB
08:08:57.182 [testcontainers-netty-1-6] ERROR ResultCallbackTemplate - Error during callback
com.github.dockerjava.api.exception.DockerException: {"message":"authorization denied by plugin pipelines: Command not supported."}

	at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:111) [testcontainers-1.7.2.jar:na]
	at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:33) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) [testcontainers-1.7.2.jar:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.211 sec <<< FAILURE!

Fabric docker plugin had the same issue and they were able to fix it with relatively small changes in plugin. See issues:

This is blocker issue and can be easily fixed.

@4ntoine 4ntoine changed the title Not working in Bitbucket pipeline. Not working in Bitbucket pipeline: "authorization denied by plugin pipelines: Command not supported." May 17, 2018

@4ntoine

This comment has been minimized.

Copy link
Author

4ntoine commented May 17, 2018

testcontainers 1.7.2 used

@kiview

This comment has been minimized.

Copy link
Member

kiview commented May 17, 2018

Hey @4ntoine, thank for the feedback regarding Bitbucket pipeline.
I'm not sure yet it it's easy to fix, our aux container (Ryuk) needs to mount the docker socket and from a quick look at the linked discussion, this is forbidden in Bitbucket pipelines.

@nathanburrell

This comment has been minimized.

Copy link

nathanburrell commented May 18, 2018

Hey @kiview test-containers works in pipelines, the issue above (as I described here: https://bitbucket.org/site/master/issues/15844/bug-in-authorization-plugin#comment-45561539) is not an issue in the auth plugin interaction with the custom java docker client nor an issue with how the library talks to docker (its working fine over tcp) its in how the MySqlContainer class by default mounts the mysql.conf file.

Users will (in order for this to work on pipelines), need to reconfigure the default (hostPath) directory for this file as we limit what paths can be mounted into docker containers due to the layout of how the containers and filesystems interact, and than the library works fine :)

@kiview

This comment has been minimized.

Copy link
Member

kiview commented May 18, 2018

Hi @nathanburrell, thanks a lot for the clarification.
So how should we ideally tackle this? Change our MySqlContainer implementation, or mention it in the docs as a workaround for pipelines compatibility?

@kiview kiview added type/bug and removed type/feature labels May 18, 2018

@4ntoine

This comment has been minimized.

Copy link
Author

4ntoine commented May 18, 2018

I believe paths should be adjusted to be in allowed by Bitbucket, see debug on local machine:
2018-05-19 0 05 00

4ntoine added a commit to 4ntoine/testcontainers-java that referenced this issue May 18, 2018

Issue testcontainers#700 - Not working in Bitbucket pipeline: "author…
…ization denied by plugin pipelines: Command not supported."

Detecting Bitbucket build and creating temporary files/dirs within BITBUCKET_CLONE_DIR.
@kiview

This comment has been minimized.

Copy link
Member

kiview commented Jun 8, 2018

Hey @4ntoine, would you like to submit your fix as a PR? We can look into it together.
Whoops sorry, you've already submitted the PR in #704

@4ntoine

This comment has been minimized.

Copy link
Author

4ntoine commented Jun 8, 2018

sorry i did not have time to fix, but i was able to use https://github.com/palantir/docker-compose-rule instead (it works)

@dstepanov

This comment has been minimized.

Copy link

dstepanov commented Aug 6, 2018

Can you provide a way to disable Ryuk?
I have found workaround:

public abstract class TestBase {
    static {
        try {
            DockerClientFactory.instance().client();
        } catch (DockerException e) {
            if (!e.getMessage().contains("BITBUCKET_CLONE_DIR")) {
                throw new IllegalStateException(e);
            }
            // Ignore exception related to reach outside of BITBUCKET_CLONE_DIR in ResourceReaper
        }
    }
}

It's ignoring first exception caused by Ryuk trying to connect to docker.sock. Is there a way to not use sock interactions?

@bsideup

This comment has been minimized.

Copy link
Member

bsideup commented Aug 6, 2018

Hi @dstepanov,

DIsabling Ryuk doesn't fix the issue, rather delays it. Does it work if you disable the startup checks? See here how to do that: https://www.testcontainers.org/usage/properties.html

@dstepanov

This comment has been minimized.

Copy link

dstepanov commented Aug 6, 2018

@nathanburrell

This comment has been minimized.

Copy link

nathanburrell commented Aug 9, 2018

Hi,

Im a developer on bitbucket pipelines, since ryuk can only communicate with a docker daemon over its unix socket currently (and the fact that pipelines only allows communication with the steps docker daemon over TCP), we have since relaxed the mounts allowed in pipelines to allow mounting in the unix socket for the steps docker daemon, which now allows ryuk to run correctly :)

Kind Regards,
Nathan Burrell

@erikvanzijst

This comment has been minimized.

Copy link

erikvanzijst commented Aug 9, 2018

now allows ryuk to run correctly :)

Note that the MySqlContainer probably still needs work if it uses a mount point for the mysql-default-conf file (the PostgreSQLContainer doesn't do this and already works correctly in Pipelines).

@bsideup

This comment has been minimized.

Copy link
Member

bsideup commented Aug 9, 2018

@nathanburrell
That is... Amazing! 😍 Thanks for listening and relaxing it from your side, I'm sure there are other projects that will also benefit from it 👍

@bsideup

This comment has been minimized.

Copy link
Member

bsideup commented Aug 9, 2018

@erikvanzijst We're working on mount-less file sharing (with copyFileToContainer with InputStream on container creation), it should fix the issue you mentioned. Will be released very soon :)

@4ntoine

This comment has been minimized.

Copy link
Author

4ntoine commented Aug 9, 2018

@nathanburrell Awesome, thanks!

@dstepanov

This comment has been minimized.

Copy link

dstepanov commented Aug 9, 2018

@nathanburrell Great news! Thanks!

@dstepanov

This comment has been minimized.

Copy link

dstepanov commented Sep 27, 2018

@nathanburrell As of today's morning versions testcontainers fail on Bitbucket pipelines with:

containers-ryuk] o.testcontainers.utility.ResourceReaper  : Can not connect to Ryuk at localhost:32768
    java.net.ConnectException: Connection refused (Connection refused)
@dstepanov

This comment has been minimized.

Copy link

dstepanov commented Sep 27, 2018

@nathanburrell Older versions fails with old error:

    com.github.dockerjava.api.exception.DockerException: {"message":"authorization denied by plugin pipelines: -v only supports $BITBUCKET_CLONE_DIR and its subdirectories"}
@bsideup

This comment has been minimized.

Copy link
Member

bsideup commented Sep 27, 2018

@dstepanov is it with 1.9.1?

@dstepanov

This comment has been minimized.

Copy link

dstepanov commented Sep 27, 2018

@bsideup 1.9.0 and 1.9.1

@nathanburrell

This comment has been minimized.

Copy link

nathanburrell commented Sep 27, 2018

Hi,

We rolled out a change yesterday to enable user namespace remapping on the docker daemons we provide to users for security reasons.

Thats an odd failure but, with this change I would expect to see

  • network=host is no longer allowed (we have added customers to an exclusion list so that they wont have user namespace remapping yet, we plan on reaching out to them over the coming weeks to have them modify their builds to either use a custom network or simple port mappings, as there is no real need for network=host).
  • permission issues with certain mounted files, for the repository we set it to already be fully open to cater for build containers using different uids to the clone and docker in docker containers, so I expect this to occur rarely.

Have you got a link to a reproducible test case showing this, or alternatively can you raise a support ticket at https://support.atlassian.com/contact, so I can investigate further.

Kind Regards,
Nathan Burrell

@dstepanov

This comment has been minimized.

Copy link

dstepanov commented Sep 28, 2018

@nathanburrell Why wouldn't you include a simple testcontainers project to test your changes? Now all our builds are down because of this.

@nathanburrell

This comment has been minimized.

Copy link

nathanburrell commented Sep 28, 2018

I understand your frustration when we make changes like this and I apoligise, but unfortunately security fixes may sometimes break some builds for the good of all the users of pipelines.

For test containers we don't test it specifically as we have lots of users that use 1000s of different libraries from a broad range of different languages that perform similar functionality and we can't test all the things unfortunately.

I haven't had a chance to look into why test containers is failing yet due to other urgencies, and as I listed above I'm surprised a connection to local host is failing as I expect only the two cases above to fail.

For now I have added your account to the exclusion list to allow you to work, next time can you raise a support ticket with Atlassian directly so we can back to you in a more timely manner as I also don't monitor GitHub issues alot only in my spare time :)

Kind Regards
Nathan Burrell

@bsideup

This comment has been minimized.

Copy link
Member

bsideup commented Sep 28, 2018

Hi @nathanburrell! Thanks for checking it at all 👍
Let me know if I can help anyhow to debug what happened, I hope that's not something big and we can solve it either on our or your side 🤞

@nathanburrell

This comment has been minimized.

Copy link

nathanburrell commented Sep 28, 2018

From looking at the github issued as well it looks like test containers or ryuk has an issue with docker usernamespace remapping as this user has reported the same error but their not running it in pipelines but locally with user namespace remapping enabled.

#712

@nathanburrell

This comment has been minimized.

Copy link

nathanburrell commented Sep 28, 2018

Also I was doing some testing with test-containers today for one of the atlassian repositories who uses this library and we found that version prior to 1.9.0 dont work with pipelines due to the pre flight checks that older versions performed having binds that we dont allow in pipelines for security reasons.

If you use a version >= 1.9.0 everything works :)

This change fixed it: 8acee4d#diff-563759d72d9839fe713305a5b9ee5d1c

And was released in 1.9.0 :)

@remmeier

This comment has been minimized.

Copy link

remmeier commented Dec 1, 2018

being able to disable ryuk would be great. For many cases this should be sufficient since the cloud provider will cleanup by itself. Even if it does not solve the problem completely, it would at least allow to work around it.

@remmeier

This comment has been minimized.

Copy link

remmeier commented Dec 6, 2018

is there an interest in a PR for such a property? For our use cases we don't really need ryuk.

@rnorth

This comment has been minimized.

Copy link
Member

rnorth commented Dec 15, 2018

I've been trying to get up to speed on this issue, as seen here and in #712. Sorry for only recently looking into this. To summarise my understanding so far:

  • User namespace remapping will always prevent access to the Docker daemon unix socket. Among other things that's part of the security benefits of the remapping.
  • As such, even a workaround of making Ryuk (and other things) talk to a TCP socket for the docker daemon is moot - even if possible it would be sidestepping the security model. Is this accurate?
  • Right now Bitbucket pipelines has an exception to allow Testcontainers to work, but this won't be for much longer.
  • For other teams using namespace remapping there is no such exception, so we have a problem.
  • Both of these cases are the same problem, it's just that we have a temporary reprieve for Bitbucket pipelines.

I'd like to think carefully about the options; disabling Ryuk might be one way of avoiding access from TC-spawned to the docker daemon. However, it's less desirable because it is not the only feature that will suffer (Docker Compose is another). Also, it creates yet another permutation for Testcontainers to operate in, one which lacks a lot of the safeguards for cleanup that we think are important. I don't feel comfortable yet with the implications of that.

Still, this is probably something that @kiview, @bsideup and I need to have a discussion about soon.

@nathanburrell

This comment has been minimized.

Copy link

nathanburrell commented Jan 10, 2019

Hey :)

Any updates on this, as we are planning on removing the exclusion list in the next month from pipelines.

Kind Regards,
Nathan Burrell

@raulgomis

This comment has been minimized.

Copy link

raulgomis commented Jan 24, 2019

Hey @rnorth ,

Any updates on this? Have you guys discussed it internally (cc: @kiview @bsideup)?

Having an opt-out parameter to disable Ryuk would work for all CI/CD use cases (like Bitbucket Pipelines). At the moment we have two sets of customers impacted:

  • Existing users using testcontainers: they will be impacted when we remove them from the exclusion list in February.
  • New users: currently being impacted as they can't use testcontainers with Bitbucket Pipelines.

I would appreciate your help here, so that we reduce the impact to our customers and they can keep using testcontainers happily in Bitbucket Pipelines.

Cheers,
Raul

@rnorth

This comment has been minimized.

Copy link
Member

rnorth commented Jan 24, 2019

Hi Raul, Nathan,
I’m very sorry for not replying sooner. Yes, we have had internal discussions and agree that disabling the Ryuk component via an environment variable is the best path forward.

@bsideup is working on it and we’d like to get this released very soon and communicated out to users.

Thank you for your help, and I apologise again.

Richard

Sent with GitHawk

@kiview

This comment has been minimized.

Copy link
Member

kiview commented Jan 24, 2019

Hi @raulgomis,
we've discussed this together in our call yesterday and we decided to solve this with #1023.

This will be a quite non invasive hack/fix and most probably won't impact the Testcontainers UX on Bitbucket Pipelines negatively, since the containers should be removed by CI environment automatically after the build.

@raulgomis

This comment has been minimized.

Copy link

raulgomis commented Jan 24, 2019

Great news!

Thanks a lot team 🙌, our users will love that you guys are planning to fix it.

Cheers,
Raul

@bsideup

This comment has been minimized.

Copy link
Member

bsideup commented Jan 25, 2019

Hi @raulgomis & @nathanburrell,

I just pushed a PR:
#1181

If you set TESTCONTAINERS_RYUK_DISABLED environment variable to true, we will not start Ryuk:
https://5c4c37a5560ab600080292aa--testcontainers.netlify.com/features/configuration/#disabling-ryuk

Please try it with Jitpack (revision 03a5c368ae) if you have time:
https://www.testcontainers.org/jitpack_dependencies/

@schrieveslaach

This comment has been minimized.

Copy link

schrieveslaach commented Jan 25, 2019

@bsideup, I tried to use your PR and now I can start containers with a docker setup providing container isolation with a user namespace mapping (see #712).

@bsideup

This comment has been minimized.

Copy link
Member

bsideup commented Jan 25, 2019

@schrieveslaach thanks for trying!

FYI the userns parameter will be supported once we integrate #1007

@schrieveslaach

This comment has been minimized.

Copy link

schrieveslaach commented Jan 25, 2019

@bsideup, thanks for pointing out this PR. I was going to ask you exactly that question. 😉

@rnorth

This comment has been minimized.

Copy link
Member

rnorth commented Jan 30, 2019

We've now released this in Testcontainers 1.10.6!

Please upgrade as soon as you can and follow the steps here for disabling Ryuk: https://www.testcontainers.org/features/configuration/#disabling-ryuk

I'll close this ticket now, but please keep us posted if there are any new issues with this.

@rnorth rnorth closed this Jan 30, 2019

@modye

This comment has been minimized.

Copy link

modye commented Jan 30, 2019

Thanks a lot, this is working like a charm in Bitbucket 😄

@nathanburrell

This comment has been minimized.

Copy link

nathanburrell commented Jan 30, 2019

Thanks for actioning this quickly :)

We are organising some communicuations to customers in the coming weeks to tell them to update and set that variable, and I can also continue to start using test-containers more in the pipelines codebase itself ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment