mount-buildkite-agent & empty directory

[NorseGaud]

I've got the following

  - label: ":centos: [CentOS] 7 Ensure Dependency Image Exists"
    command:
      - "$APT_PREP"
      # - "mkdir -p ~/.ssh && chmod 600 ~/.ssh && ssh-keyscan -H github.com >> ~/.ssh/known_hosts"
      - "$ENSURE_LINUX_IMAGE centos:7 $IMAGE_TAG_VERSION"
    agents:
      queue: "automation-eos-builder-fleet"
    plugins:
      - docker#v3.2.0:
          image: "ubuntu:18.04"
          shell: ["/bin/bash", "-i", "-e", "-c"]
          debug: true
    timeout: 120
    skip: $SKIP_CENTOS_7

Buildkite is showing that running as

docker run -it --rm --init --volume /home/kubernetes/flexvolume/buildkite/builds/buildkite-k8s-agents-eos-builder-588d697f99-wgml2-1/EOSIO/base-images-test:/workdir --workdir /workdir --env BUILDKITE_JOB_ID --env BUILDKITE_BUILD_ID --env BUILDKITE_AGENT_ACCESS_TOKEN --volume /usr/local/bin/buildkite-agent:/usr/bin/buildkite-agent ubuntu:18.04 /bin/bash -i -e -c apt-get update && apt-get upgrade -y && apt-get install -y git sudo jq

However, I ls -laht /usr/bin/buildkite-agent and see an empty directory.

Any ideas what's wrong?

[toolmantim]

Hey @NorseGaud! Sorry for not spotting this earlier.

Does /usr/local/bin/buildkite-agent exist on the host (if you run a command with ls -lsa /usr/local/bin/buildkite-agent)? And I wonder if it's being disallowed to be bind-mounted into the container, or that's being blocked somehow?

[NorseGaud]

No problem, sorry for the delay. I got pulled into an urgent issue. Here are the results without mount-buildkite-agent specified in our pipeline.yml:

ls -laht /usr/local/bin/buildkite-agent; ls -laht /usr/bin/buildkite-agent

ls: cannot access /usr/local/bin/buildkite-agent: No such file or directory
--
  | total 8.0K
  | dr-xr-xr-x 1 root root 4.0K Jul 25 17:31 ..
  | drwxr-xr-x 2 root root 4.0K Jul 24 19:50 .

Checking on the bind-mount permissions.

[NorseGaud]

Ok, I just ran a test using volumes and got the following:

          volumes:
            - "/usr/local/bin/buildkite-agent"

total 8.0K
--
  | drwxr-xr-x 1 root root 4.0K Jul 25 17:38 ..
  | drwxr-xr-x 2 root root 4.0K Jul 25 17:38 .
  | total 8.0K
  | dr-xr-xr-x 1 root root 4.0K Jul 25 17:38 ..
  | drwxr-xr-x 2 root root 4.0K Jul 24 22:19 .

So even volumes is mounting an empty directory instead of the file.

[NorseGaud]

Aha! On my local machine, same:

command -v buildkite-agent
/usr/local/bin/buildkite-agent

docker run -ti --rm -v $(pwd):/workdir -v /usr/local/bin/buildkite-agent eosio/producer:eosio-ubuntu-18.04-97c66ab9db7382be04dea9e9
598fd110d66432f1 bash
root@4a86d4037865:/# command -v buildkite-agent
root@4a86d4037865:/# ll /usr/local/bin/
total 56676
drwxr-xr-x 1 root root     4096 Jul 25 17:41 ./
drwxr-xr-x 1 root root     4096 Jul 25 13:23 ../
drwxr-xr-x 2 root root     4096 Jul 25 17:41 buildkite-agent/
-rwxr-xr-x 1 root root 18833304 Jul 25 13:23 cmake*
-rwxr-xr-x 1 root root 18844128 Jul 25 13:23 cpack*
-rwxr-xr-x 1 root root 20312080 Jul 25 13:23 ctest*
-rwxr-xr-x 1 root root    24720 Jul 25 13:26 mongoc-stat*

[NorseGaud]

Furthermore, mounting the dir where it is doesn't work either:

docker run -ti --rm -v $(pwd):/workdir -v /usr/local/bin:/host-bin eosio/producer:eosio-ubuntu-18.04-97c66ab9db7382be04dea9e9598fd110d66432f1 bash
root@44a3b6b66446:/# ls -laht /host-bin/
total 158M
drwxr-xr-x 1 root root 4.0K Jul 25 17:43 ..
drwxr-xr-x 2 root root 2.0K Feb 13 10:29 .
-rwxr-xr-x 1 root root  27M Feb 10 04:14 containerd
-rwxr-xr-x 1 root root 4.8M Feb 10 04:14 containerd-shim
-rwxr-xr-x 1 root root  15M Feb 10 04:14 ctr
-rwxr-xr-x 1 root root  49M Feb 10 04:14 docker
-rwxr-xr-x 1 root root 747K Feb 10 04:14 docker-init
-rwxr-xr-x 1 root root 2.8M Feb 10 04:14 docker-proxy
-rwxr-xr-x 1 root root  52M Feb 10 04:14 dockerd
-rwxr-xr-x 1 root root 7.2M Feb 10 04:14 runc

If I ls -laht /usr/local/bin on the host I get a ton more stuff.

Found docker/for-win#25 which is for windows...

Update: Just realized we're doing docker inside of docker. So it doesn't exist to mount. Not a plugin issue!

[NorseGaud]

For anyone interested in a solution... We installed buildkite-agent in the dockerfile and just used the --token and --agent-access-token to do start or upload commands in the various scripts. We did have to pass in the right ENVs which were available on the first docker layer:

environment:
  - "BUILDKITE_AGENT_ACCESS_TOKEN" # Needed for buildkite-agent upload
mount-buildkite-agent: false # Mounting bk-agent doesn't work, so disable it
propagate-environment: true # Need for buildkite-agent upload (JOB_ID, etc)