Cache miss when running in container

[snobu]

Consider this simple multi-stage build:

FROM debian:stretch-slim as build

ENV DENO_VERSION 0.1.9
ENV DENO_DIR /build
RUN mkdir -p /build

RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates \
    curl \
    && apt-get clean -y \
    && rm -rf /var/lib/apt/lists/* \
    && curl -fsSLO --compressed "https://github.com/denoland/deno/releases/download/v$DENO_VERSION/deno_linux_x64.gz" \
    && gunzip -c deno_linux_x64.gz > /build/deno \
    && chmod u+x /build/deno \
    && rm deno_linux_x64.gz

COPY hello.ts /build
WORKDIR /build
RUN ./deno hello.ts


FROM debian:stretch-slim
COPY --from=build /build/deno /app/deno
COPY --from=build /build/deps /app/deps
COPY --from=build /build/gen /app/gen
COPY --from=build /build/hello.ts /app/
WORKDIR /app
ENV DENO_DIR /app
CMD ./deno hello.ts

The expectation is deno's compile cache will be hit when the resulting image is launched.
However, this doesn't happen:

$ docker run hellodeno
Compiling /app/hello.ts
Compiling https://gist.githubusercontent.com/ry/f12b2aa3409e6b52645bc346a9e22929/raw/79318f239f51d764384a8bded8d7c6a833610dde/print_hello.ts
--- Hello from hello.ts ---

$ docker run hellodeno
Compiling /app/hello.ts
Compiling https://gist.githubusercontent.com/ry/f12b2aa3409e6b52645bc346a9e22929/raw/79318f239f51d764384a8bded8d7c6a833610dde/print_hello.ts
--- Hello from hello.ts ---

# RUN ./deno -D hello.ts

$ docker run hellodeno 2>&1 | grep -i cache
DEBUG RS - load_cache /app/gen/4e4584381ac19c16f9e1c6ae4db535f7521c99bd.js
DEBUG RS - load_cache /app/gen/790f0b07cf14ce8eda5eaab841b453acaf2e4da3.js
DEBUG JS - os.ts codeCache /app/hello.ts // tslint:disable-next-line:max-line-length
DEBUG RS - msg_from_js CodeCache sync true
DEBUG JS - os.ts codeCache /app/deps/gist.githubusercontent.com/ry/f12b2aa3409e6b52645bc346a9e22929/raw/79318f239f51d764384a8bded8d7c6a833610dde/print_hello.ts export function printHello(): void {
DEBUG RS - msg_from_js CodeCache sync true

^^ Not sure how to read that, some cache was loaded but not used?

If i exec (docker run -it image_name bash) inside the container, then the second time i run ./deno hello.ts, the cache is hit, so i guess DENO_DIR works.

There seems to be some other state outside deps and gen that i'm failing to copy over. Where's that state?

I see some calls to /dev/urandom with strace, if that's saved somewhere and then reused.. it could explain the cache miss.

[snobu]

Also, is there a way to just bundle up everything a la Golang static builds? One static ELF binary with everything in so you can FROM scratch your way into Prod.

[njbraun]

Any update on this would be appreciated. I'm running into the exact same issue. (cached source code is missed, and new hashes are generated for all sources. aka, all of DENO_DIR/gen is regenerated.)

Looks like the method that generates the cache_key only takes 3 params:

fn source_code_hash(
  filename: &str,
  source_code: &[u8],
  version: &str,
) -> String {
  let mut ctx = ring::digest::Context::new(&ring::digest::SHA1);
  ctx.update(version.as_bytes());
  ctx.update(filename.as_bytes());
  ctx.update(source_code);
  let digest = ctx.finish();
  let mut out = String::new();
  // TODO There must be a better way to do this...
  for byte in digest.as_ref() {
    write!(&mut out, "{:02x}", byte).unwrap();
  }
  out
}

Now the question is, which one of the 3 is changing?

[ry]

Can you try printing the arguments of source_code_hash ?

Maybe this has something to do with a redirect happening during resolution.

If you could provide the source code you're trying to run that would help.

[bartlomieju]

@njbraun there were some changes to source_code_hash recently, can you check if this issue is still happening with latest release?

[snobu]

Unfortunately, it behaves the same for me, cache miss, deno 0.5.0 -

$ docker run hellodeno
[1/2] Compiling https://gist.githubusercontent.com/ry/f12b2aa3409e6b52645bc346a9e22929/raw/79318f239f51d764384a8bded8d7c6a833610dd
[2/2] Compiling https://gist.githubusercontent.com/ry/f12b2aa3409e6b52645bc346a9e22929/raw/79318f239f51d764384a8bded8d7c6a833610dde/print_hello.ts
--- Hello from hello.ts ---


$ docker run hellodeno
[1/2] Compiling https://gist.githubusercontent.com/ry/f12b2aa3409e6b52645bc346a9e22929/raw/79318f239f51d764384a8bded8d7c6a833610dd
[2/2] Compiling https://gist.githubusercontent.com/ry/f12b2aa3409e6b52645bc346a9e22929/raw/79318f239f51d764384a8bded8d7c6a833610dde/print_hello.ts
--- Hello from hello.ts ---

[bartlomieju]

Thanks for checking, I'll try to debug the problem

[bartlomieju]

@snobu I've replicated your setup - the problem is ENV DENO_DIR /build. You are using directory inside container as cache directory (DENO_DIR). Every time you run container this directory is created and there's no cache. Try mounting volume from host machine and using it as cache directory - then the problem goes away.

FROM debian:stretch-slim as build

ENV DENO_VERSION 0.5.0
ENV DENO_DIR /denodir
RUN mkdir -p /build
WORKDIR /build
RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates \
    curl \
    && apt-get clean -y \
    && rm -rf /var/lib/apt/lists/* \
    && curl -fsSLO --compressed "https://github.com/denoland/deno/releases/download/v$DENO_VERSION/deno_linux_x64.gz" \
    && gunzip -c deno_linux_x64.gz > /build/deno \
    && chmod u+x /build/deno \
    && rm deno_linux_x64.gz

COPY hello.ts /build

ENTRYPOINT ["/build/deno", "run", "-D", "hello.ts"]

Then:

docker build -t hellodeno .
docker run -v `pwd`/container_cache:/denodir hellodeno

Let me know if you need anymore info

[hayd]

You can also put a deno fetch hello.ts inside a RUN block (after the COPY).

I think a nice pattern for this is:

COPY deps.ts /build
RUN deno fetch deps.ts  # cache the deps (unless you change them)
COPY . /build
RUN deno fetch app.ts  # compile, so there is no compile step on startup

ENTRYPOINT ["deno", "run", "app.ts"]

[zekth]

Also you're not forced to mount it to your machine but you can keep it in docker volumes. Check the doc: https://docs.docker.com/storage/volumes/

[snobu]

Thank you @bartlomieju. That actually makes perfect sense, somehow i completely forgot how containers work since i opened this issue.

May i suggest an official Dockerfile example be added to the repo with a note on caching (persistent volume)? I'm sure there are others like me out there :)

[hayd]

I pushed this deps.ts to my deno_docker example (deps are cached as a layer so don't need to be refetched + compiled unless deps.ts changes, this speeds up the Docker dev cycle significantly).

https://github.com/hayd/deno_docker/tree/master/example

[ry]

I’d be happy to have a Dockerfile as long as it was exercised in CI.

[zekth]

https://github.com/hayd/deno_docker/tree/master/example

Providing a docker-compose would be great too with sharing of cache between containers. Like

version: "3"

services:
  deno2:
    image: deno2
    volumes:
       - deno-cache:/cache/
    ports:
      - "8080:80"

  deno1:
    image: deno1
    volumes:
       - deno-cache:/cache/
    ports:
      - "8090:80"

volumes:
  deno-cache:

[hayd]

@zekth I guess that should work OOTB if you specify DENO_DIR?
I haven't used volumes myself, but happy to add an example/mention somewhere if it's useful.

[hayd]

@ry I suspect you want these in a sep repo so that you can test images against release binaries (rather than generate your own). I think adding to CI would be kinda fiddly, if you wanted to keep the image small, as you wouldn't be able to cache layers... at least that's my understanding, perhaps there's an easy way that's also fast.

[zekth]

@zekth I guess that should work OOTB if you specify DENO_DIR?
I haven't used volumes myself, but happy to add an example/mention somewhere if it's useful.

You may have multiple cases. One is to mount your cache on your host another is to mount the cache in docker volumes. Really depends on the case / uses. It's the same has having npm install inside the dockerfile or having it prepared before packaging the docker image.

In the case of your example you fetch only main.ts, if the script is for example app.ts it does not fetch it, but having it mounted on a docker volume or host volume will prevent this fetch.

I can't be really sure about all the scenarios, i think it will be after several uses we will figure out what's the best solution.

[hayd]

@zekth Let's move that discussion to https://github.com/hayd/deno_docker/issues/4 (or a new issue). I think your example could/would work already.

[bartlomieju]

This issue was resolved, see comment.

I suppose we can close this issue now, CC @ry