Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stage1 caching #1494

Merged
merged 22 commits into from
Sep 12, 2018
Merged

stage1 caching #1494

merged 22 commits into from
Sep 12, 2018

Conversation

andrewrk
Copy link
Member

@andrewrk andrewrk commented Sep 10, 2018

See #1416

  • move builtin.zig to its own cache
  • get rid of zig-cache dir
  • output files into cwd
  • fix zig build not knowing where the output paths will be
  • fix tests
  • include .h files from translate-c in hash (error message for attempting @cImport and --cache on for now)
  • macos os.cpp implementations
  • windows os.cpp implementations
  • include libc in hash (error message for attempting --library c --cache on for now)
  • cache eviction (upon successful stage2 build we will delete the stage1 cache)
  • tmp file cleanup

see #1344
closes #881

from master branch of blake2 reference implementation
320c325437539ae91091ce62efec1913cd8093c2
 * add almost all the input parameter state to the hash
   - missing items are the detected MSVC installation on Windows
     and detected libc installation on POSIX
   - also missing are C files and .h files that libclang finds
 * artifacts are created in global cache directory instead of
   zig-cache.
   - exception: builtin.zig is still in zig-cache
 * zig run uses the new cache correctly
 * zig run uses execv on posix systems
@andrewrk andrewrk added the work in progress This pull request is not ready for review yet. label Sep 10, 2018
@andrewrk andrewrk added this to the 0.3.0 milestone Sep 10, 2018
@andrewrk
Copy link
Member Author

Sadly we can't use hard links to output artifacts, because users could - even though they shouldn't - modify the output files, which would undermine the assumptions of the cache system. I shudder to think of the bug reports I would get if this happened.

I'm thinking that we do one of two things:

  1. copy output files from the cache dir to the output path. This has the unfortunate downside that if the compilation is never done again, this was a waste of a file copy. If the artifact is very large, this could be quite expensive, and potentially even if it was a cache hit, could be more expensive than creating the output file in place.
  2. make a distinction between different kinds of builds. In practice, the caching system will help when building compiler_rt.a, builtin.a, zig run, and zig build, and usually won't help otherwise. We always know which one we're doing, so it could just be enabled for the high reward ones and disabled otherwise. We'd never need to copy from cache dir to output directory - in all 4 cases above we can use the artifacts directly from the cache dirs. And then normal builds can output artifacts directly to their final output paths (either cwd or the explicitly provided --output-file)

After typing this out, it seems like (2) would be preferable.

...zig run, zig build, compiler_rt.a, and builtin.a
@ghost
Copy link

ghost commented Sep 11, 2018

does this make the tests run faster or something like that? because in the one issue you liked above it sais

Since this is a large, fairly complicated and only impacts usability/performance of the command, this should only be implemented for the self-hosted compiler.

@andrewrk
Copy link
Member Author

andrewrk commented Sep 11, 2018

There are several problems causing CI failures right now. One of them is that we hit the limit for how long tests can run. This shaves off about 10% of the time.

Also I got tired of zig build taking a long time when working on self-hosted :)

we need somewhere to put .o files and leave them while the user
executes their program, so that stack traces on MacOS can find
the .o files and get at the DWARF info.

if we try to clean up old global tmp dir files, first of all that's
a hard and complicated problem, and secondly it's not clear how
that is better than dumping the .o file inside zig-cache locally.
and have it print llvm's internal timing info
@andrewrk andrewrk removed the work in progress This pull request is not ready for review yet. label Sep 12, 2018
@andrewrk andrewrk changed the title WIP: stage1 caching stage1 caching Sep 12, 2018
@andrewrk andrewrk merged commit 0cfd019 into master Sep 12, 2018
@andrewrk andrewrk deleted the stage1-caching branch September 12, 2018 16:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add caching to run command
1 participant