Fix docker caching by resetting cache only when mono changes #38972
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chsienki
force-pushed
the
fix_dockerfile_caching
branch
2 times, most recently
from
77d86ac
to
46fa171
Compare
chsienki
force-pushed
the
fix_dockerfile_caching
branch
from
46fa171
to
be70772
Compare
|
Note also: because we have machine fan out, this change will take a little while to have any effect. As the new docker file is used on each machine it will start from scratch (as the file changed), and then subsequent runs on the same machines should improve in speed as the cache is used. |
|
@dotnet/roslyn-infrastructure for review please. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We currently have a 'cache bust' mechanism, that prevents our docker image from caching the mono nightly install steps: We cache the initial install, then update the container every time we run the tests to get up to latest.
In general this is fine, but as the base image gets further away from the 'latest' update, the apt-upgrade takes longer and longer, leading to machine restore time being up to 50% of overall test time.
We have a couple of options, including not updating everything on the box, but this can lead to potentially weird issues where a 'fresh' container build would give different results to a cached one due to dependency differences, which we definitely want to avoid.
This change breaks the container build into two stages: an info stage and a build stage. The info stage simply gets the release info for Mono nightly. The main build then copies this file into the main container before updating. Docker is clever enough to use the hash of the file to determine if it should maintain the cache or not, meaning we'll only update the machine and its dependencies if a new nightly mono has actually been published since the last run.
There are still some issues with this approach, mainly that running on different physical machines can end up with different dependency sets cached, but they should only ever be slightly out of date, so I'm hopeful we shouldn't see any actual issues.
I have some ideas how to fix this in a much more robust way, but it needs some thinking / infra investment, so this should be considered a medium term fix to try and get the Linux Mono legs to a better, but not perfect, state.