Problem running tests on Azure Pipelines

[wimdeblauwe]

I have a few tests that use testcontainers-keycloak that work just fine locally (on macOS). However, on Azure Pipelines the tests fail. The tests are for a custom token mapper. From what I can see, I think the extensions are not loaded (The test fail on the fact that the custom token that my extension adds is not present), although I load them using:

        keycloak = new KeycloakContainer("quay.io/keycloak/keycloak:15.0.1")
                .withRealmImportFile("keycloak-test-realm.json")
                .withExtensionClassesFrom("target/classes");

I enabled DEBUG logging and I do see this:

Cmd: org.testcontainers.shaded.com.github.dockerjava.core.command.CreateContainerCmdImpl@1ea9f009[name=<null>,hostName=<null>,domainName=<null>,user=<null>,attachStdin=<null>,attachStdout=<null>,attachStderr=<null>,portSpecs=<null>,tty=<null>,stdinOpen=<null>,stdInOnce=<null>,env={KEYCLOAK_PASSWORD=admin,DB_VENDOR=h2,KEYCLOAK_IMPORT=/tmp/keycloak-test-realm.json,KEYCLOAK_USER=admin},cmd={-c standalone.xml,-b 0.0.0.0,-Dkeycloak.profile.feature.upload_scripts=enabled},healthcheck=<null>,argsEscaped=<null>,entrypoint=<null>,image=quay.io/keycloak/keycloak:15.0.1,volumes=com.github.dockerjava.api.model.Volumes@5d52e3ef,workingDir=<null>,macAddress=<null>,onBuild=<null>,networkDisabled=<null>,exposedPorts=com.github.dockerjava.api.model.ExposedPorts@5298dead,stopSignal=<null>,stopTimeout=<null>,hostConfig=HostConfig(binds=[/azp/_work/1/s/target/test-classes/../../target/classes:/opt/jboss/keycloak/standalone/deployments/-495401560-extensions.jar:rw,Z], blkioWeight=null, blkioWeightDevice=null, blkioDeviceReadBps=null, blkioDeviceWriteBps=null, blkioDeviceReadIOps=null, blkioDeviceWriteIOps=null, memorySwappiness=null, nanoCPUs=null, capAdd=null, capDrop=null, containerIDFile=null, cpuPeriod=null, cpuRealtimePeriod=null, cpuRealtimeRuntime=null, cpuShares=null, cpuQuota=null, cpusetCpus=null, cpusetMems=null, devices=null, deviceCgroupRules=null, deviceRequests=null, diskQuota=null, dns=null, dnsOptions=null, dnsSearch=null, extraHosts=[], groupAdd=null, ipcMode=null, cgroup=null, links=[], logConfig=LogConfig(type=null, config=null), lxcConf=null, memory=null, memorySwap=null, memoryReservation=null, kernelMemory=null, networkMode=null, oomKillDisable=null, init=null, autoRemove=null, oomScoreAdj=null, portBindings={}, privileged=null, publishAllPorts=true, readonlyRootfs=null, restartPolicy=null, ulimits=null, cpuCount=null, cpuPercent=null, ioMaximumIOps=null, ioMaximumBandwidth=null, volumesFrom=[], mounts=null, pidMode=null, isolation=null, securityOpts=null, storageOpt=null, cgroupParent=null, volumeDriver=null, shmSize=null, pidsLimit=null, runtime=null, tmpFs=null, utSMode=null, usernsMode=null, sysctls=null, consoleSize=null),labels={org.testcontainers=true, org.testcontainers.sessionId=cd46a8a4-1380-496c-96b2-9ab441febe61},shell=<null>,networkingConfig=<null>,ipv4Address=<null>,ipv6Address=<null>,aliases=<null>,authConfig=<null>]

Especially this part I think is good:

hostConfig=HostConfig(binds=[/azp/_work/1/s/target/test-classes/../../target/classes:/opt/jboss/keycloak/standalone/deployments/-495401560-extensions.jar:rw,Z]

I also have this in the logging:

2021-11-02 07:33:40 DEBUG [ducttape-0] WaitingConsumer - STDOUT: 07:33:39,407 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 43) WFLYSRV0010: Deployed "keycloak-server.war" (runtime-name : "keycloak-server.war")
2021-11-02 07:33:40 DEBUG [ducttape-0] WaitingConsumer - STDOUT: 07:33:39,405 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 32) WFLYSRV0010: Deployed "-495401560-extensions.jar" (runtime-name : "-495401560-extensions.jar")

So it looks like the extension is loaded. Is there anything else I can look at to further debug this?

Note that on Azure, I am using a custom build agent running in Docker on an Ubuntu host. The custom build agent is started like this:

docker run -d --name build-agent-java-1 -v /var/run/docker.sock:/var/run/docker.sock -e AZP_URL=https://dev.azure.com/my-organisation -e AZP_TOKEN=<mytoken> -e AZP_AGENT_NAME=agent-1 custombuildagentcr.azurecr.io/ubuntu1804-java11

I have -v /var/run/docker.sock:/var/run/docker.sock there to make sure testcontainers works inside the Docker container.

[wimdeblauwe]

According to https://www.testcontainers.org/supported_docker_environment/continuous_integration/dind_patterns/, I need to use -v $PWD:$PWD -w $PWD as well when starting the container of the build agent. But that does not work with the need of Azure DevOps to have /azp as the working directory and calling start.sh as entrypoint (See https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops#linux).

Reading through the answers of https://stackoverflow.com/questions/31381322/docker-in-docker-cannot-mount-volume also leads me to believe that the "inner docker" that runs Keycloak cannot access the sources of the "outer docker" that is the build agent on Azure.

There is also this documentation: Mounting volumes using Docker within a Docker container, but I don't quite understand if and how this applies to what I am trying to do.

[dasniko]

Unfortunately I don't have any experience neither with Azure nor with Docker-in-Docker...

[wimdeblauwe]

Ok, thanks. I posed a question on stackoverflow as well: https://stackoverflow.com/questions/69810959/testcontainers-with-volume-mounts-on-custom-azure-devops-build-agent-running-in

[dasniko]

Hi @wimdeblauwe,
over the weekend, I released v1.9.0 of the Keycloak Testcontainers (based on KC16 as default KC version).
Now, the extensions are no more just "bind mounted" from the source directory to the container, but actually copied to it, see #44

If you still have the issue, please test it with the most recent version. Perhaps this is related to this.

[wimdeblauwe]

Ah, very nice. I'll definitely try this version. I hope it still works with Keycloak 15.0.1 as we are currently on that one.

[wimdeblauwe]

Just tested and it works perfectly with this version. Thanks!

[luedi]

I have a similar issue, but in the opposite order. I'm using the testcontainer-keycloak ina local environment on Windows with Docker-Desktop runninm g on WSL2. With version 1.7.1 everything works perfect, but when i change to the current version 1.9.0 my SPI is not found:

org.testcontainers.containers.ContainerLaunchException: Container startup failed

	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:336)
...
Caused by: java.lang.IllegalArgumentException: Resource with path target\classes\xxx\xxx\tokenmapper\MyCustomAttributeMapper.class could not be found on any of these classloaders: [jdk.internal.loader.ClassLoaders$AppClassLoader@368239c8]
...

My startup code is:

        try (KeycloakContainer keycloak = new KeycloakContainer("jboss/keycloak:16.1.0")
                .withExtensionClassesFrom("target/classes")) {
            keycloak.start();

When I debug the keycloak startup i notice that createKeycloakExtensionDeployment use an absolute path to my SPI class file (C:/Projekte/.../target/classes) in version 1.7.1 while using a relative path (.../target/classes) in version 1.9.0. This is caused by the change you mentioned above (replacing addFileSystemBind).
I'm not sure if this is a problem in the library or if i have to provide additional configuration to make my tests work. I'm working with IntelliJ IDEA and use the standard unittest runner (just start the test from within the editor).

[dasniko]

Please don't mention "similar" behavior under a thread with a different topic, it gets lost and/or ignored. Opening a new thread with referencing the "similar" thing would be better.

Please test your issue with most recent version from master branch, I just merged #48 which might address your issue. Thanks.