New feature request: Selectively disable caching for specific RUN commands in Dockerfile

[mohanraj-r]

branching off the discussion from #1384 :

I understand -no-cache will disable caching for the entire Dockerfile. But would be useful if I can disable cache for a specific RUN command? For example updating repos or downloading a remote file .. etc. From my understanding that right now RUN apt-get update if cached wouldn't actually update the repo? This will cause the results to be different than from a VM?

If disable caching for specific commands in the Dockerfile is made possible, would the subsequent commands in the file then not use the cache? Or would they do something a bit more intelligent - e.g. use cache if the previous command produced same results (fs layer) when compared to a previous run?

[tianon]

I think the way to combat this is to take the point in the Dockerfile you do want to be cached to and tag that as an image to use in your future Dockerfile's FROM, that can then be built with -no-cache without consequence, since the base image would not be rebuilt.

[mohanraj-r]

But wouldn't this limit interleaving cached and non-cached commands with ease ?

For e.g. lets say I want to update my repo and wget files from a server and perform bunch of steps in between - e.g. install software from the repo (that could have been updated) - perform operations on the downloaded file (that could have changed in the server) etc.

What would be ideal is for a way to specify to docker in the Dockerfile to run specific commands without cache every time and the only reuse previous image if there is no change (for e.g no update in repo).

Wouldn't this be useful to have ?

[wagerlabs]

What about CACHE ON and CACHE OFF in the Dockerfile? Each instruction would affect subsequent commands.

[konklone]

Yeah, I'm using git clone commands in my Dockerfile, and if I want it to re-clone with updates, I need to, like, add a comment at the end of the line to trigger a rebuild from that line. I shouldn't need to create a whole new base container for this step.

[githart]

Can a container ID be passed to 'docker build' as a "do not cache past this ID" instruction? Similar to the way in which 'docker build' will cache all steps up to a changed line in a Dockerfile?

[shykes]

I agree we need more powerful and fine-grained control over the build cache. Currently I'm not sure exactly how to expose this to the user.

I think this will become easier with the upcoming API extensions, specifically naming and introspection.

[timruffles]

Would be a great feature. Currently I'm using silly things like RUN a=a some-command, then RUN a=b some-command to break the cache

[rogernolan]

Getting better control over the cache would make using docker from CI a lot happier.

[crosbymichael]

@shykes

What about changing --no-cache from a bool to a string and have it take a regex for where in the docker we want to bust the cache?

docker build --no-cache "apt-get install" .

[shykes]

I agree and suggested this exact feature on IRC.

Except I think to preserve reverse compatibility we should create a new flag (say "--uncache") so we can keep --cached as a (deprecated) bool flag that resolves to "--uncache .*"

On Fri, Feb 7, 2014 at 9:17 AM, Michael Crosby notifications@github.com
wrote:

@shykes
What about changing --no-cache from a bool to a string and have it take a regex for where in the docker we want to bust the cache?

docker build --no-cache "apt-get install" .

Reply to this email directly or view it on GitHub:
dotcloud#1996 (comment)

[crosbymichael]

What does everyone else think about this? Anyone up for implementing the feature?

[timruffles]

I'm up for having a stab at implementing this today if nobody else has started?

[timruffles]

I've started work on it - wanted to validate the approach looks good.

• The noCache field of buildfile becomes a *regexp.Regexp.
  • A nil value there means what utilizeCache = true used to.
• Passing a string to docker build --no-cache now sends a validate regex string to the server.
• Just calling --no-cache results in a default of .*
• The regex is then used in a new method buildfile.utilizeCache(cmd []string) bool to check commands that ignore cache

One thing: as far as I can see, the flag/mflag package doesn't support string flags without a value, so I'll need to do some extra fiddling to support both --no-cache and --no-cache some-regex

[tianon]

I really think this ought to be a separate new flag. The behavior and syntax of --no-cache is already well defined and used in many, many places by many different people. I'd vote for --break-cache or something similar, and have --no-cache do exactly what it does today (since that's very useful behavior that many people rely on and still want).

Anyways, IANTM (I am not the maintainer) so these are just my personal thoughts. :)

[timruffles]

@tianon --no-cache is currently bool, so this simply extends the existing behaviour.

• docker build --no-cache - same behaviour as before: ignores cache
• docker build --no-cache someRegex - ignores any RUN or ADD commands that match someRegex

[tianon]

Right, that's all fine. The problem is that --no-cache is a bool, so the existing behavior is actually:

• --no-cache=true - explicitly disable cache
• --no-cache=false - explicitly enable cache
• --no-cache - shorthand for --no-cache=true

I also think we'd be doing ourselves a disservice by making "true" and "false" special case regex strings to solve this, since that will create potentially surprising behavior for our users in the future. ("When I use --no-cache with a regex of either 'true' or 'false', it doesn't work like it's supposed to!")

[timruffles]

@tianon yes you're right. Had a quick look and people are using =true/false.

Happy to modify the PR to add new flag as you suggest, what do the maintainers think (@crosbymichael, @shykes)? This would also mean I could remove the code added to mflag to allow string/bool flags.

[crazyscience]

+1 for @wagerlabs approach

[marcuslinke]

@crosbymichael, @timruffles Wouldn't it be better if the author of the Dockerfile decides which build step should be cached and which should not? The person that creates the Dockerfile is not necessarily the same that builds the image. Moving the decision to the docker build command demands detailed knowledge from the person that just want to use a specific Dockerfile.

Consider a corporate environment where someone just want to rebuild an existing image hierarchy to update some dependencies. The existing Dockerfile tree may be created years ago by someone else.

[hunterloftis]

+1 for @wagerlabs approach

[cressie176]

+1 for @wagerlabs approach although it would be even nicer if there was a way to cache bust on a time interval too, e.g.

CACHE [interval | OFF]
RUN apt-get update
CACHE ON

I appreciate this might fly against the idea of containers being non deterministic, however it's exactly the sort of thing you want to do in a continuous deployment scenario where your pipeline has good automated testing.

As a workaround I'm currently generating cache busters in the script I use to run docker build and adding them in the dockerfile to force a cache bust

FROM ubuntu:13.10
ADD ./files/cachebusters/per-day /root/cachebuster
...
ADD ./files/cachebusters/per-build /root/cachebuster
RUN git clone git@github.com:cressie176/my-project.git /root/my-project

[tfoote]

I'm looking to use containers for continuous integration and the ability to set timeouts on specific elements in the cache would be really valuable. Without this I cannot deploy. Forcing a full rebuild every time is much too slow.

My current plan to work around this is to dynamically inject commands such as RUN echo 2014-04-17-00:15:00 with the generated line rounded down to the last 15 minutes to invalidate cache elements when the rounded number jumps. ala every 15 minutes. This works for me because I have a script generating the dockerfile every time, but it won't work without that script.

[amarnus]

+1 for the feature.

[hiroprotagonist]

I also want to vote for this feature. The cache is annoying when building parts of a container from git repositories which updates only on the master branch.
👍

[amarnus]

@hiroprotagonist Having a git pull in your ENTRYPOINT might help?

[hiroprotagonist]

@amarnus I've solved it similar to the idea @tfoote had. I am running the build from a jenkins job and instead of running the docker build command directly the job starts a build skript wich generates the Dockerfile from a template and adds the line 'RUN echo currentsMillies' above the git commands. Thanks to sed and pipes this was a matter of minutes. Anyway, i still favor this feature as part of the Dockerfile itself.

[dannykansas]

Adding my +1 for @wagerlabs approach. Also having this issue with CI. I'm simply using a dynamic echo RUN statement for the time being, but I would love this feature.

[leonardschneider]

+1 for CACHE ON/OFF. My use case is also CI automation.

[stilliard]

+1, especially the ability to set a run commands cache interval like in @cressie176 's example

[disposable-ksa98]

"For example updating repos or downloading a remote file"

+1

[dannykansas]

If it helps anyone, here's the piece of code I'm using in my Jenkins build:

echo "Using build $BUILD_NUMBER for docker cachebusting"
sed -i s/cachebust_[0-9]*/cachebust_"$BUILD_NUMBER"/g Dockerfile

[bfitzsimmons]

+1 for CACHE ON/OFF

[claytondaley]

As a possible alternative to the CACHE ON/OFF approach, what about an extra keyword like "ALWAYS". The keyword would be used in combination with an existing command (e.g. "ALWAYS RUN" or "ALWAYS ADD"). By design, the "ALWAYS" keyword does not go to the cache to complete the adjacent command. However, it compares the result to the CACHE (implicitly the cache for other times the same line was executed), linking to the cached image if the result of the ALWAYS command is unchanged.

I believe the underlying need is to identify "non-idempotent instructions". The ALWAYS command does this very explicitly. My impression is that the CACHE ON/OFF approach could work equally well, but could aso require lots of switching over blocks of code (which may encourage users to block off more lines than really required).

[kuon]

I am also more for a prefix to commands, like ALWAYS or CACHE 1 WEEK ADD ...

[CheRuisiBesares]

So I was struggling with this issue for a while and I just wanted to share my work around incase its helpful while this gets sorted out. I really didn't want to add anything outside of the docker file to the build invocation or change the file every time. Anyway this is a silly example but it uses the add mechanism to bust the cache and doesn't require any file manipulations.

From ubuntu:14.04

RUN apt-get -yqq update
RUN apt-get -yqq install git
RUN git clone https://github.com/coreos/fleet
ADD http://www.random.org/strings/?num=10&len=8&digits=on&upperalpha=on&loweralpha=on&unique=on&format=plain&rnd=new uuid
RUN cd fleet && git pull

Obviously you can pick your own use case and network random gen. Anyway maybe it will help some people out idk.

[gzankevich]

Another +1 for @wagerlabs approach

[assertrandom]

Another +1 to the feature. Meanwhile using @cruisibesarescondev workaround.

[tcarlyle]

one more +1 for the feature request. And thanks to @cruisibesarescondev for the workaround

[toldjuuso]

Another +1 for the feature.

Cheers @cruisibesarescondev for the workaround.

[tfoote]

I think the ALWAYS keyword is a good approach, especially as it has simple clear semantics. A slightly more complicated approach would be to add a minimum time, (useful in things like a buildfarm or continuous integration). For that I'd propose a syntax "EVERY XXX" where XXX is a timeout. And if it's been longer than XXX since the cache of that command was built it must rerun the command. And check if the output has changed. If no change reuse the cached result, noting the last updated time. This would mean that EVERY 0 would be the same as ALWAYS.

For a workaround at the moment I generate my Dockerfiles using empy templates in python and I embed the following snippets which works as above except that does not detect the same result in two successive runs, but does force a retrigger every XXX seconds. At the top:

@{
import time
def cache_buster(seconds):
    ts = time.time()
    return ts - ts % seconds
}@

Where I want to force a rerun:

RUN echo @(cache_buster(60))

Which looks like this in the Dockerfile

RUN echo 1407705360.0

As you can see it rounds to the nearest 60 so each time 60 seconds pass the next run will rerun all following commands.

[pikeas]

+1 for ALWAYS syntax. +.5 for CACHE ON/CACHE OFF.

[hellais]

+1 for ALWAYS syntax.

[kigiri]

Yes, ALWAYS syntax looks very intuitive.

[kuon]

I don't like CACHE ON/OFF because I think lines should be "self contained" and adding blocks to Dockerfiles would introduce a lot of "trouble" (like having to check "is this line covered by cache?" when merging...).

[thaJeztah]

@kuon I think there are already a number of commands that affect subsequent instructions, e.g. USER and WORKDIR

[kuon]

Yeah, that's true, but I don't use them for the same reason. I always do RUN cd ... && or RUN su -c ...&&.

I'd prefer a block notation:

CACHE OFF {
    RUN ...
}

This is more explicit and avoid having a CACHE OFF line inserted by mistake (it would trigger a syntax error).

I might be overthinking it, Dockerfiles are not actually run in production (just when building the image), so having the cache disabled when you build won't actually do much harm. But I also feel Dockerfiles are really limiting (having to chain all commands with a && in a single RUN to avoid creating a gazillion of images, not being able to use variables...).

Maybe this issue is the opportunity for a new Dockerfile format.

[kuon]

I'd like to come back on what I just said. I read what @shykes said in another issue docker#2266 and I also agree with him (Dockerfile need to stay a really simple assembly like language).

I said I'd like variable or things like that, but that can be covered by some other language, but in this case, each line in a Dockerfile should be self contained, eg:

NOIMAGE ALWAYS RUN USER:jon  apt-get update

Which would always run the command (no cache), but would also not create an image and use the user jon.

This kind of self contained line are much easier to generate from any other language. If you have to worry about the context (user, cache, workdir), it's more error prone.

[ghost]

Can it be RUN! for ease, please?

[abramsm]

Any status update on this one?

[orrery]

Selectively disabling the cache would be very useful. I grab files from a remote amazon s3 repository via the awscli command (from the amazon AWS toolkit), and I have no easy way to bust the cache via an ADD command (at least I can't think of a way without editing the Dockerfile to trigger it). I believe there is a strong case for control to be given back to the user to selectively bust the cache when using RUN. If anyone has a suggestion for me I'd be happy to hear from you.

[hellais]

Wanted to bump this issue up a bit since it's something that we have a big need for.

Still convinced ALWAYS syntax is the ideal one.

[cpuguy83]

How about a simple BREAK statement.

[hellais]

@cpuguy83 that would work also for my particular use case.

I am not sure if it's technically possible to have only one command not be cached, but the rest of them to be cached. Probably not since docker is based on incremental diffs.

Having support for BREAK though would give me feature parity with my current workaround based on the suggestion by @CheRuisiBesares.

[orrery]

Regarding my previous post, it would indeed be sufficient to just bust the cache from that point in the script onwards, the rest would just be down to intelligent script design (and I believe this would address most people's requirements). Is this doable instead of selectively disabling cache bust?

[RyanHartje]

So many +1s, if you pull the git repo in your docker file, cache keeps your images from building. Makes it kind of hard to push builds through CI.

[Vingtoft]

+1 cloning git repos (its very annoying that the image needs to be build from scratch each time a small edit has been made in a git repo)

[itsprdp]

@Vingtoft If you are updating the files in the repo then your cache is invalidated.

[Vingtoft]

@itsprdp I did not know that, thank you for clarifying.

[Vingtoft]

@itsprdp I have just tested. When I'm updating the repo and building the image, Docker is still using the cache.
Perhaps I'm misunderstanding something?

[RyanHartje]

@itsprdp That isn't correct in my experience. I made a new commit to a repo to test, and when building again, it uses the same cache.

If I change the docker file previous to the repo, of course it will be cache busted, however simply updating a repo does not seem to fix this issue.

[itsprdp]

@RyanHartje Sorry for the confusion. It is supposed to invalidate the cache if the repository is updated and that's something to consider by contributors.
The use case @Vingtoft expecting is to cache the repository and only update the changed files in the repository. This might be complicated to implement.

[Vingtoft]

@itsprdp Only updating the changed files in a repo would be awesome, but less (or should I say more?) would do as well.
In my use-case (and many others) the actual git pull does not take long time: Its the building of everything else thats killing the development flow.

[douineauromain]

+1, cache used during git clone :(

[shane-axiom]

An integrated solution would be nice, but in the meantime you can bust the cache at a specific Dockerfile instruction using ARG.

In the Dockerfile:

ARG CACHEBUST=1
RUN git clone https://github.com/octocat/Hello-World.git

On the command line:

docker build -t your-image --build-arg CACHEBUST=$(date +%s) .

Setting CACHEBUST to the current time means that it will always be unique, and instructions after the ARG declaration in the Dockerfile won't be cached. Note that you can also build without specifying the CACHEBUST build-arg, which will cause it to use the default value of 1 and preserve the cache. This can be used to always check out fresh copies of git repos, pull latest SNAPSHOT dependencies, etc.

Edit: Which, uh, is just what @thaJeztah said. I'll leave this up as an additional description of his solution.

[pulkitsinghal]

@shane-axiom How about using the git commit hash as the value for CACHEBUST?

export CACHEBUST=`git ls-remote https://username@bitbucket.org/username/myRepo.git | grep refs/heads/develop | cut -f 1` && \
echo $CACHEBUST && \
docker build -t myDockerUser/myDockerImage \
   --build-arg blah=blue \
   --build-arg CACHEBUST=$CACHEBUST \
   .

Based on clues from http://stackoverflow.com/questions/15677439/how-to-get-latest-git-commit-hash-command#answer-15679887

[shane-axiom]

@pulkitsinghal That looks wonderful for busting the cache for git repos. For other uses (such as pulling in SNAPSHOT dependencies, etc) the always-busting timestamp approach works well.

[recursionbane]

+1 for CACHE ON | OFF

[pavel64-sutyrin]

+1

[KBoehme]

+1

[nikow]

Remember about @CheRuisiBesares aproach, you can always use ADD https://www.random.org/strings/?num=16&len=16&digits=on&upperalpha=on&loweralpha=on&unique=on&format=plain&rnd=new uuid as workaround for cache issues.

[brycereynolds]

To post an additional use-case....

COPY package.json /usr/src/
RUN npm install

In our package.json we will commonly point to a master tag for some of our private github dependencies. This means that we never really get the latest master unless we change the package.json file (usually just add to the description a - to the description then remove it while testing).

A RUN NO CACHE to take place of RUN seems like it would be a good solution.

[mmobini]

+1

I have similar issue for npm install which use cache and dont use my new published library in npm.

It would be great if I can disable cache per RUN command in docker file.

[thaJeztah]

@brycereynolds @mmobini see docker#1996 (comment) for manually busting the cache. However, not specifying a specific version of packages that need to be installed may not be best practice, as the end-result of your Dockerfile (and source code) is no longer guaranteed to be reproducible (i.e., it builds successfully today, but doesn't tomorrow, because one of the packages was updated). I can see this being "ok" during development, but for production (and automated builds on Docker Hub), the best approach is to explicitly specify a version. Doing so also allows users to verify the exact packages that were used to produce the image.

[ctrimble]

I have a use case where not being able to invalidate the cache is causing issues. I am running Dropwizard applications (Java REST Services built with Maven) from Docker and an automated system is doing all of the container builds and deployment for me. I include a Dockerfile in my repo and it does the rest. The system runs a production version and one or more development versions of my application. Development builds are where I am having issues.

During development, some of the project's dependencies have SNAPSHOT in their version numbers. This instructs Maven that the version is under development and it should bring down a new version with every build. As a result, an identical file structure can result in two distinct builds. This is the desired behavior, since bugs may have been fixed in a SNAPSHOT dependency. To support this, it would be helpful to force Docker to run a particular command, since there is no way to determine the effect of the command based on the current state of the file system. A majority of Java projects are going to run into this, since Maven style SNAPSHOT dependencies are used by several different build systems.

[cpuguy83]

@ctrimble You can use --no-cache, or --build-arg to invalidate the cache.
You can minimize the effect of --no-cache by having a base image with all the cacheable commands.

[ctrimble]

@cpuguy83 thank you for the reply. I read the thread and understand the current options. I have opened a ticket with the build system I am using to supply a cache busting argument. Producing two distinct images for a single application seems like a lot of hoops to go through to speed up builds. It would be much easier to be able to specify something like:

1. do things that can be cached if the file system is identical
2. do a thing that might change the file system based on when it is executed
3. do some more things that could be cached if the previous step did not change the file system

This pattern will come up in development builds frequently. It would be nice to have semantics for it in the Dockerfile.

[cpuguy83]

@ctrimble Busting the cache on one step will cause the cache to always be busted for each subsequent step.

[ctrimble]

@cpuguy83 exactly. The semantics of my build system are temporal for development builds. I have to select correct builds over caching. I would really like to get both.

[atrauzzi]

There's been considerable discussion here, apologies if it's already been suggested, but what if there was something like this:

CHECK [FILE_PATH]

All docker would do is store the MD5 (or whatever other hash is hip) of the file and if it changes, all steps thereafter are invalidated.

I'd probably be doing something like:

CHECK Gemfile
CHECK package.json
CHECK composter.json
CHECK project.json

May also want to enable a check that some how elapses after a time period. Ansible's cache_valid_time parameter for the apt plugin might offer some inspiration: http://docs.ansible.com/ansible/apt_module.html

For that, the syntax would be:

EXPIRE 1234567 
RUN apt-get update
RUN bundle install

Docker would know the last-run time and calculate if the time had elapsed based on "now".

[cpuguy83]

@atrauzzi We just support --squash on build now in 1.13 (experimental only for now).

[atrauzzi]

@cpuguy83 Are there any docs or explanations about --squash anywhere that I can read up on? At the outset the name doesn't make it sound like it does what I'm thinking. But I could be (and most likely am) wrong!

[cpuguy83]

@atrauzzi yes, in the build reference.

Basically, --squash both preserves the layer cache, and creates a 2nd image which is as if everything in the Dockerfile happened in a single layer.

[cpuguy83]

I don't see why one would need to check that a file cache is still valid individually, ADD and COPY already do this for everything that's being copied in.

[atrauzzi]

@cpuguy83 Good point, didn't even think that, and of course I'm already using it.

What about the timestamp/duration approach? Is that doable with what's already available?

[thaJeztah]

What about the timestamp/duration approach? Is that doable with what's already available?

Through build-args;

ARG expire_after=never
RUN do some thing

docker build --build-arg expire_after=2016-12-01 -t foo .

change the build arg to bust the cache

[sarpk]

+1 for a cleaner way

[ianseyer]

+1 for a cleaner way

[multi-io]

There should also be separate options for disabling reading the cache and for disabling writing to it. For example, you may want to build an image anew from scratch and ignore any cached layers, but still write the resulting new layers to the cache.

[jrusk]

+1

[chris13524]

Might I suggest passing the step number to the build command?

Something like this:
docker build --step 5 .

It would ignore all caches after and including step 5 during the build.

[bupadon]

+1
Please.

[neoxue]

CACHE ON|OFF +1

[chris13524]

The issue with these CACHE ON|OFF commands is that at whatever step the cache is turned off, there is no way to cache further steps. The only sensible command would be ENDCACHE.

[dreamcat4]

It is a valid idea / ethos. The command is supposed to coalese together all non-cached layers into a single layer at the point when the cache gets switched back on. Of course you can still argue the best naming / correctness of semantics / preferred syntax of the feature.

[MrCheater]

+1

[CageFox]

+1 the must have feature

[StalkAlex]

Agree for CACHE ON|OFF +1

[BeauAnasson]

+1 Would be amazing.

[lxblvs]

I did not really understand the way Docker caches the steps before and spent half a day investigating why my system is not building correctly. It was the "git clone" caching.

Would love to have the ALWAYS keyword.

[ChipmunkV]

How it's closed?

What is the best workaround?

[habeebr]

I tried #1996 (comment) and it worked
In the Dockerfile:

ARG CACHEBUST=1
RUN git clone https://github.com/octocat/Hello-World.git

On the command line:

docker build -t your-image --build-arg CACHEBUST=$(date +%s)

[stints]

Why not create a new command similar to RUN but doesn't ever cache RUNNC for RUN NO CACHE?

[lumannnn]

I can confirm, @habeebr (#1996 (comment)) - I use it in combination with #1996 (comment)

[naoko]

+1

[andrepuschmann]

RUNNC is a great idea!

[dolphy01]

Why was this issue closed? Between the myriad duplicates asking for essentially the same thing and the lengthy comment history of more than one of these duplicates, it seems obvious that there is a healthy interest in seeing this functionality available.

I get that it's hard, and perhaps that no one has suggested a sufficiently elegant solution that both meets the need and is clean enough be an attractive Docker addition...but that does not mean that there is no need.

The only other argument I've heard in favor of closing this is that there are other ways to accomplish this...but that argument doesn't really pass muster either. Creating multiple base images for the sole purpose of getting around the lack of cache control is unwieldy, contriving an invalidation through an ARG is obtuse and unintuitive. I imagine users want to utilize these "workarounds" about as much as Docker developers want to officially incorporate a sloppy hack into the tool.

[duglin]

its not hard: #10682
easy solution, easy UX. Just no clear consensus on whether it should be done.