Don't mutate pkg dir, use cache dir instead (and put artifacts in it) #68
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
maleadt
force-pushed
the
tb/cache
branch
5 times, most recently
from
2cf9942
to
f40fe66
Compare
|
Paging @staticfloat: any idea where these failures come from? Basically, when adding artifact caching by mounting '.cache/NewPkgEval/artifacts' to '.julia/artifacts' in the container, several JLLs fail to install when ading Gtk: Xorg_libxkbfile on 1.3.0
Fontconfig on 1.5
PCRE on 1.3
at_spi2_atk on 1.5
Could this be due to non-atomic changes to the artifact store? But this happens when installing, in parallel, Gtk, Crayons, Example and TimerOutputs. Only Gtk requires these JLLs, so I don't see where the races could come from (I don't think Travis would automaticalily share home folders across jobs?). If this approach is fundamentally flawed, any suggestions for alternative approaches to cache artifacts across NewPkgEval jobs? I basically want to avoid a 500MB download for every package that depends on CuArrays. |
I know I could use a PkgServer for this, but that may be a little early to build PkgEval on (not supporting older versions of Julia), and I'd like to avoid the need to orchestrate multiple containers. Maybe I just have to, though. Do we have some infra for a simple caching PkgServer? |
Yes, see https://github.com/JuliaPackaging/PkgServer.jl/tree/master/deployment
Hmmmm. Unsure. It seems more likely to me that this is a download issue than that its an unpacking issue. Is this only happening on 1.3, or does it happen on 1.4 as well? |
I don't test 1.4 here, but it happens both on 1.3 and master. And it definitely has something to do with sharing |
|
Ah, are you installing multiple copies of the same artifact at once? I suppose that is possible, but it should be fairly atomic; things are downloaded and unpacked into randomly-named directories, then the directory is moved to the final location after the treehash is verified. The only thing I can think of is that this |
maleadt
force-pushed
the
tb/cache
branch
3 times, most recently
from
921e5b7
to
cd0480a
Compare
|
Never mind, redefining |
maleadt
force-pushed
the
tb/cache
branch
2 times, most recently
from
e01a0bc
to
6d0631d
Compare
Closed
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #41, fixes #42