pin-docker-images2 failing with src version 3.23.3

[malomarrec]

Crash message
❌ 3 errors:
github.com/sourcegraph-testing/docker-image-pin-2:
run: find . -name Dockerfile -type f |
(... and 2 more lines)
container: sourcegraph/dockerlint-run:latest
standard error:
# ./Dockerfile:1:1: latest-image-tag
open ./Dockerfile: permission denied
Command failed with exit code 123.
Log: /tmp/changeset-github.com-sourcegraph-testing-docker-image-pin-2.234954162.log

Steps to reproduce

• Install src scli version 3.23.3
• Create a campaign with this spec
• Set publish to false
• Run src campaigns preview

[mrnugget]

I can reproduce the issue locally and ran git bisect to find which commit introduced it and git bisect tells me it's #412.

I can confirm that by running src campaign apply with -workspace bind: that works. -workspace volume doesn't work.

[mrnugget]

Maybe it's something with the sourcegraph/dockerlint-run:latest image? If I change the steps from this

  - run: |
      find . -name Dockerfile -type f |\
      xargs dockerlint -w
    container: sourcegraph/dockerlint-run:latest

to this

  - run: |
      find . -name Dockerfile -type f |\
      xargs cat
    container: alpine:3

it works, even though they both open the Dockerfile?

[mrnugget]

Okay, @eseliger and I are investigating. What we've got so far:

run	container	workspace-mode	works
rm -rf Dockerfile	sourcegraph/dockerlint-run	volume	❌
rm -rf Dockerfile	sourcegraph/dockerlint-run	bind	✅
rm -rf Dockerfile	alpine:3	volume	✅
rm -rf Dockerfile	alpine:3	bind	✅

[mrnugget]

The cause for the error is that the sourcegraph/dockerlint-run image uses UID 1000 but the owner of the /work/* files created in the volume by the workspace creator are owned by root.

We tried to hack around this by passing --user $UID:$UID to the unzip container, by calling chown $UID:$UID /work before unzipping, but... seems like permissions and Docker volumes are a bit tricky. The permissions are somehow lost after the chown and unzip with --user $UID:$UID:

Setting chown without a temp file in volume

1. Create volume:

$ VOL=$(docker volume create)
$ echo $VOL
37cf3ef795fb9c482bef4c07eb9742b62edbc06c477dbdbf52955d3759c8bb45

2. Set chown:

$ docker run  --mount type=volume,source=$VOL,target=/work \
  --rm --init sourcegraph/src-campaign-volume-workspace \
  chown -R 1000:1000 /work

3. Check chown stats (note: with --user 1000:1000):

$ docker run \
  --mount type=volume,source=$VOL,target=/work \
  --rm \
  --init --user 1000:1000 \
  sourcegraph/src-campaign-volume-workspace \
  stat -c '%u %g' /work
1000 1000

4. Fail to unzip (note: with --user 1000:1000):

$ docker run \
  --mount type=volume,source=$VOL,target=/work \
  --rm \
  --init \
  --workdir /work --user 1000:1000 \
  --mount type=bind,source=/Users/thorstenball/tmp/docker-image-pin-2.zip,target=/tmp/zip,ro \
  sourcegraph/src-campaign-volume-workspace \
  unzip /tmp/zip
Archive:  /tmp/zip
91c6f7d1fdd18ca53d13a20ea331cb2414e804b5
error:  cannot create Dockerfile
        Permission denied
error:  cannot create README.md
        Permission denied
error:  cannot create package.json
        Permission denied
error:  cannot create prettier2.1-2.2-updates.js
        Permission denied

5. Check chown stats (note: with --user 1000:1000):

$ docker run \
  --mount type=volume,source=$VOL,target=/work \
  --rm \
  --init --user 1000:1000 \
  sourcegraph/src-campaign-volume-workspace \
  stat -c '%u %g' /work
0 0

Setting chown WITH a temp file in volume

But somewhere in a GitHub issue I can't find now, someone said something about empty volumes losing permissions, so we put a stupid empty file in the volume before the unzip... and... guess what?

1. Create volume

$ VOL=$(docker volume create)                                                                                                                                                                   -130-
$ echo $VOL
37cf3ef795fb9c482bef4c07eb9742b62edbc06c477dbdbf52955d3759c8bb45

2. Set chown:

$ docker run  --mount type=volume,source=$VOL,target=/work \
  --rm --init sourcegraph/src-campaign-volume-workspace \
  chown -R 1000:1000 /work

3. Create stupid file (note: as root):

$ docker run  --mount type=volume,source=$VOL,target=/work \
  --rm --init sourcegraph/src-campaign-volume-workspace \
  touch /work/DELETE_ME

4. Check ownership (note: with --user 1000:1000):

$ docker run \
  --mount type=volume,source=$VOL,target=/work \
  --rm \
  --init --user 1000:1000 \
  sourcegraph/src-campaign-volume-workspace \
  stat -c '%u %g' /work
1000 1000

5. Unzip *successfully (note: with --user 1000:1000):

$ docker run \
  --mount type=volume,source=$VOL,target=/work \
  --rm \
  --init \
  --workdir /work --user 1000:1000 \
  --mount type=bind,source=/Users/thorstenball/tmp/docker-image-pin-2.zip,target=/tmp/zip,ro \
  sourcegraph/src-campaign-volume-workspace \
  unzip /tmp/zip
Archive:  /tmp/zip
91c6f7d1fdd18ca53d13a20ea331cb2414e804b5
  inflating: Dockerfile
  inflating: README.md
  inflating: package.json
  inflating: prettier2.1-2.2-updates.js

6. Check stats (note: with --user 1000:1000):

$ docker run \                                                                                                                                                                                   -50-
  --mount type=volume,source=$VOL,target=/work \
  --rm \
  --init --user 1000:1000 \
  sourcegraph/src-campaign-volume-workspace \
  stat -c '%u %g' /work
1000 1000

---

We created a ridiculous prototype that fixes this issue here: #433

But the word "ridiculous" should be underlined too.