WITH DOCKER does not get CACHE mounts

[rrjjvv]

Example:

VERSION --use-cache-command 0.6

with:
    FROM earthly/dind:alpine
    CACHE /some/dir
    RUN echo "yo" > /some/dir/greet
    WITH DOCKER
        RUN --no-cache ls -al /some/dir
    END

$ earthly --version
earthly version v0.6.30 743687991d227f2f7e2e6ba11a90f443b4f99c67 linux/amd64; Ubuntu 22.04.1 LTS (Jammy Jellyfish)

$ earthly -P +with
<snip>
 earthly/dind:alpine | --> Load metadata linux/amd64
               +with | --> FROM earthly/dind:alpine
               +with | [██████████] 100% resolve docker.io/earthly/dind:alpine@sha256:9b09804c68a2c68195b71e61a09616657b8f0a5ca01b4e029f710c7ca2396dc6
               +with | --> RUN echo "yo" > /some/dir/greet
               +with | --> WITH DOCKER (install deps)
               +with | /usr/bin/jq
               +with | /usr/local/bin/dockerd
               +with | --> WITH DOCKER RUN --privileged --no-cache ls -al /some/dir
               +with | Starting dockerd with data root /var/earthly/dind/234e6b660326d22f9e997893fe6b3d24513d7865067df5b47fef9840337608e8/tmp.igGKDh
               +with | total 8
               +with | drwxr-xr-x    2 root     root          4096 Jan  5 18:13 .
               +with | drwxr-xr-x    3 root     root          4096 Jan  5 18:13 ..
<snip>

My original problem was actually the reverse; making changes to the mount within WITH DOCKER were not persisted (thus not visible/usable in future invocations), but this easier to illustrate.

I can't think of a reason why this shouldn't work, so is probably a logical extension of #2331. I'm guessing it's as simple as taking

earthly/earthfile2llb/converter.go

Lines 594 to 596 in 0946ea9

	for _, cache := range c.persistentCacheDirs {
	opts.extraRunOpts = append(opts.extraRunOpts, cache)
	}

and adding it prior to any/all applicable docker variants, e.g.

earthly/earthfile2llb/withdockerrun.go

Line 240 in 0946ea9

_, err = w.c.internalRun(ctx, crOpts)

and

earthly/earthfile2llb/withdockerrunreg.go

Line 230 in 0946ea9

_, err = w.c.internalRun(ctx, crOpts)

[alexcb]

One thing to note is that the RUN --no-cache will also prevent the CACHE from being available in the container that executes the command.

In order to test the fix, I had to add a specific line that I would update between runs in order to bust the cache.

VERSION --use-cache-command 0.6

with:
    FROM earthly/dind:alpine
    CACHE /some/dir
    RUN echo "yo" > /some/dir/greet
    RUN echo "bust the cache -- iteration 8" # I would then increment this number (ctrl-A in vim)
    RUN ls -al /some/dir
    WITH DOCKER
        RUN ls -al /some/dir
    END

[rrjjvv]

Oh, that's good to know. That was not a part of my actual scenario (I very rarely use --no-cache in my Earthfiles); I only added it for the sake of an all-in-one reproduction (without having those manual steps).

Would it be more accurate to say that it's "available, but not referenced" (i.e., "write-only")? I would expect the cache to be written to (so other steps could see/use any changes), though it wouldn't use the contents itself.

Even if you truly meant unavailable (not even mounted), that's fine, just not what I would expect (relative to how the rest of earthly works).