Docker in Docker with additional_containers doesn't seem to work?

[Jackenmen]

Expected Behavior

I expected docker daemon to be exposed on port 2375 when using this configuration:

example_task:
  # also happens for `container` though that isn't as important
  # because `docker_builder` exists
  arm_container:
    image: docker:latest
    additional_containers:
      - name: dockerdaemon
        privileged: true
        cpu: 4
        memory: 3500Mb
        image: docker:dind
        port: 2375
        env:
          DOCKER_DRIVER: overlay2
  env:
    DOCKER_HOST: tcp://localhost:2375
  script: docker run ubuntu echo hello

Real Behavior

The docker daemon was not exposed and when trying to connect to it I got an error message:

docker run ubuntu echo hello
docker: Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?.
See 'docker run --help'.
Exit status: 125

Link to the task output:

• container: https://cirrus-ci.com/task/5163998269145088
• arm_container: https://cirrus-ci.com/task/5620585639182336

Related Info

This is a (tick one of the following):

• Website issue
  • Link to page:
• Task issue
  • OS: Docker (container:/arm_container:)
  • Task name: example
  • Script/cache name (if applies): main

Additional information

I've also tried a less special image such as the MySQL docker image that was presented in the documentation and it also did not seem to work:

example_task:
  container:
    image: ubuntu:latest
    additional_containers:
      - name: mysql
        image: mysql:latest
        port: 3306
        cpu: 1.0
        memory: 512Mi
        env:
          MYSQL_ROOT_PASSWORD: ""
  script:
    - apt update
    - apt install -y curl net-tools
    - netstat -tulpn
    - curl http://localhost:3306

outputs for those:

• container: https://cirrus-ci.com/task/4790913250623488
• arm_container: https://cirrus-ci.com/task/5256499583778816

[edigaryev]

Please read the TLS section of Docker image: since 18.09+, TLS connection is used by default and is available on port 2376 (not 2375).

To enforce the old behavior, pass DOCKER_TLS_CERTDIR= environment variable to the container.

[Jackenmen]

What about the MySQL repro though?

[edigaryev]

What about the MySQL repro though?

Sorry, didn't noticed that part of your message.

It seems that if you want to supply an empty password via MYSQL_ROOT_PASSWORD="", you also need to provide a non-empty MYSQL_ALLOW_EMPTY_PASSWORD environment variable, otherwise the MySQL container will exit.

[Jackenmen]

This solved the MySQL issue.

As for the Docker in Docker, the section that you referred to says to use dockerd as the entrypoint (which afaik translates to command: in Cirrus CI's container configuration) which sadly did not work. Then I also tried the solution you suggested (setting DOCKER_TLS_CERTDIR to an empty string) and that also did not work. Note that in all of these cases, I checked the output of netstat -tulpn and there was no sign of the docker daemon:

• https://cirrus-ci.com/task/5144830434082816
• https://cirrus-ci.com/task/6001597422501888

Btw, is there any way to see output from additional containers or in general know whether they work? There is some info about starting the container in debugging information but no console output or anything from that container that would allow me to figure out why it doesn't work.

---

I'd like to also say why I thought both of the examples from the issue description will just work - Cirrus CI's documentation presents both of them:

• https://cirrus-ci.org/guide/writing-tasks/#additional-containers
• https://cirrus-ci.org/guide/docker-builds-on-kubernetes/#how-to
  Could these examples be updated? It might save time for other people who are trying to reproduce examples from the documentation in their projects.

[edigaryev]

As for the Docker in Docker, the section that you referred to says to use dockerd as the entrypoint (which afaik translates to command: in Cirrus CI's container configuration) which sadly did not work.

You only need to set DOCKER_TLS_CERTDIR to an empty value, see dockerd-entrypoint.sh.

Then I also tried the solution you suggested (setting DOCKER_TLS_CERTDIR to an empty string) and that also did not work.

This is strange indeed, I'm going to reopen this for further investigation.

Could these examples be updated? It might save time for other people who are trying to reproduce examples from the documentation in their projects.

See #1035.

[Jackenmen]

Just to keep the repro simple, here are two tasks doing the DOCKER_TLS_CERTDIR: "":

• container (https://cirrus-ci.com/task/4628828566847488):

example_task:
  container:
    image: docker:latest
    additional_containers:
      - name: dockerdaemon
        privileged: true
        image: docker:dind
        port: 2375
        env:
          DOCKER_DRIVER: overlay2
          DOCKER_TLS_CERTDIR: ""
  env:
    DOCKER_HOST: tcp://localhost:2375
  script:
    - netstat -tulpn
    - docker run ubuntu echo hello

• arm_container (https://cirrus-ci.com/task/4827335143718912):

example_task:
  arm_container:
    image: docker:latest
    additional_containers:
      - name: dockerdaemon
        privileged: true
        image: docker:dind
        port: 2375
        env:
          DOCKER_DRIVER: overlay2
          DOCKER_TLS_CERTDIR: ""
  env:
    DOCKER_HOST: tcp://localhost:2375
  script:
    - netstat -tulpn
    - docker run ubuntu echo hello

[fkorotkov]

Unfortunately, we don't support privileged additional containers on managed by us clusters and dind need it. You can use it with your own clusters on GCP, AWS or Oracle cloud. For the managed instances you can use something like that:

example_task:
  container:
    image: docker:latest
    kvm: true
    additional_containers:
      - name: dockerdaemon
        image: docker:dind
        cpu: 2
        memory: 4G
        port: 2375
        env:
          DOCKER_DRIVER: overlay2
          DOCKER_TLS_CERTDIR: ""
  env:
    DOCKER_HOST: tcp://localhost:2375
  sleep_script: sleep 30 # so docker starts
  script: docker run ubuntu echo hello

[joseluisq]

@fkorotkov I'm using your setup but using a matrix. It works for the amd64 but not for the arm64. I got docker: Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?

https://cirrus-ci.com/task/6248340032061440

task:
  only_if: $CIRRUS_TAG == ''
  env:
    DOCKER_HOST: tcp://localhost:2375
  matrix:
    - name: linux-amd64
      container:
        image: docker:latest
        kvm: true
        additional_containers:
          - name: dockerdaemon
            image: docker:dind
            cpu: 2
            memory: 4G
            port: 2375
            env:
              DOCKER_DRIVER: overlay2
              DOCKER_TLS_CERTDIR: ""

    - name: linux-arm64
      arm_container:
        image: docker:latest
        kvm: true
        additional_containers:
          - name: dockerdaemon
            image: docker:dind
            cpu: 2
            memory: 4G
            port: 2375
            env:
              DOCKER_DRIVER: overlay2
              DOCKER_TLS_CERTDIR: ""

  sleep_script: sleep 30 # so docker starts
  script: docker run ubuntu echo hello

[Jackenmen]

Maybe arm containers can't be KVM-enabled? The documentation isn't clear on that and arm containers are using different services entirely (EKS Cluster on AWS rather than GKE cluster on Google Cloud) so it seems plausible. Would certainly be nice if this is something that Cirrus CI developers could fix though.

Note that you can now also just use Docker Builder VM which can be ran on Cirrus CI's community cluster:
https://cirrus-ci.org/guide/docker-builder-vm/

That's what I ended up using for cibuildwheel and it's been working well.

[joseluisq]

Note that you can now also just use Docker Builder VM which can be ran on Cirrus CI's community cluster: https://cirrus-ci.org/guide/docker-builder-vm/

That's what I ended up using for cibuildwheel and it's been working well.

Yes, I'm aware of that. it's very useful.
However, I'm not interested in arm64 qemu emulation from an amd64 via buildx because of the slowness involved.
But rather, I would like to build a multi-arch image in two separate instances. One using the Linux container (GKE cluster of compute-optimized instances) and another with the arm_container (EKS cluster of Graviton2 instances). And finally, join them together via a docker manifest.

By the way, Is that doable right now from a Cirrus standing point?

[Jackenmen]

I'm talking about native arm64, that's what the CIRRUS_ARCH env var is for (however weird that may sound). It uses Google Cloud's ARM64 VMs. You can see documentation of that by switching to arm64 tab.
Here's how we use it (we use compute_engine_instance option directly rather than docker_builder with CIRRUS_ARCH because we wanted to adjust assigned resources):
https://github.com/pypa/cibuildwheel/blob/main/.cirrus.yml#L21

[fkorotkov]

@joseluisq arm_container doesn't support privileged mode. The kvm option for container under the hood simply creates a Docker Builder VM and does docker run --privileged for all the containers including the additional_containers.

I think theoretically you should be able to build and push two manifests separately and than stitch them together like this post:

task:
  name: Push Manifest ($CIRRUS_ARCH)
  alias: manifest
  compute_engine_instance:
    image_project: cirrus-images
    image: family/docker-builder-arm64
    matrix:
      architecture: amd64
      architecture: arm64
    platform: linux
  build_script: ...
  login_script: ...
  push_script: docker push your-username/multiarch-example:manifest-$CIRRUS_ARCH

combine_docker_builder:
  login_script: ...
  create_script: |
    docker manifest create \
      your-username/multiarch-example:manifest-latest \
      --amend your-username/multiarch-example:manifest-amd64 \
      --amend your-username/multiarch-example:manifest-arm64
  push_script: docker manifest push your-username/multiarch-example:manifest-latest

[joseluisq]

@jack1142 @fkorotkov thank you guys for your help.
I already tested both compute_engine_instance tasks and work like a charm based on your setup files.
I will also do the adjustments for pushing the manifests. Which should not be a problem, I think.

Cirrus CI rocks! ⚡