New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ir2obj cache pruning support. #1753

Merged
merged 2 commits into from Sep 14, 2016

Conversation

Projects
None yet
2 participants
@JohanEngelen
Member

JohanEngelen commented Sep 11, 2016

This implements ir2obj cache pruning in the compiler itself, and also through a stand-alone tool (they share the cache pruning D implementation).

Cache pruning scheme:

  1. Check the prune interval (-ir2obj-cache-prune-interval and ldc-prune-cache --interval) to see if pruning is already needed again. A timestamp file is written to the cache for this.
  2. First remove all "expired" files, i.e. files with an age older than a set limit (-ir2obj-cache-prune-expiration and ldc-prune-cache --expiry).
  3. Then, if the cache size is still too large, remove files (oldest first) until the size is below a set limit. The limit can be specified absolutely (--ir2obj-cache-prune-maxbytes and ldc-prune-cache --max-bytes) and/or relatively (--ir2obj-cache-prune-maxpercentage and ldc-prune-cache --max-percentage-of-avail).

Pruning is done after codegen, but before linking. So the prune parameters must be such that the object files needed for linking will survive (e.g. the cache size limit should be large enough).

Pruning is off by default, but specifying any pruning parameters on the commandline implies enabling it (--ir2obj-cache-prune).
The reasoning for having it off per default is that in projects using parallel compilation threads, you'd want the pruning to happen at the end of a build cycle (using the standalone tool). Otherwise, the first finishing compilation thread may start pruning stuff that you might cache-hit in another compile thread. Also, multi-threaded pruning is untested at best.

The pruning code uses the file's "last access time". But because this is not automatically updated upon symlinking or usage by the linker on some platforms (Windows 7, from what I read online), I added an extra "touch" upon a cache hit such that the last access time is refreshed at least during compilation.

Resolves #1747

driver/ir2obj_cache_pruning.h
+#ifndef LDC_DRIVER_IR2OBJ_CACHE_PRUNING_H
+#define LDC_DRIVER_IR2OBJ_CACHE_PRUNING_H
+
+void DpruneCache(const char *cacheDirectoryPtr, size_t cacheDirectoryLen,

This comment has been minimized.

@klickverbot

klickverbot Sep 11, 2016

Member

Sorry for nit-picking, but DpruneCache is not a great name ("D"?), especially since it is hardcoded to "ircache_…".

@klickverbot

klickverbot Sep 11, 2016

Member

Sorry for nit-picking, but DpruneCache is not a great name ("D"?), especially since it is hardcoded to "ircache_…".

This comment has been minimized.

@JohanEngelen

JohanEngelen Sep 12, 2016

Member

wip !
(it was to set it apart from the previous C++ implementation)

@JohanEngelen

JohanEngelen Sep 12, 2016

Member

wip !
(it was to set it apart from the previous C++ implementation)

@JohanEngelen JohanEngelen changed the title from [wip] Add ir2obj cache pruning support. to Add ir2obj cache pruning support. Sep 13, 2016

@JohanEngelen JohanEngelen reopened this Sep 13, 2016

@JohanEngelen

This comment has been minimized.

Show comment
Hide comment
@JohanEngelen

JohanEngelen Sep 13, 2016

Member

All green :)
Don't think I can make it better than this.

Member

JohanEngelen commented Sep 13, 2016

All green :)
Don't think I can make it better than this.

+#define LDC_DRIVER_IR2OBJ_CACHE_PRUNING_H
+
+#if __LP64__
+using d_ulong = unsigned long;

This comment has been minimized.

@klickverbot

klickverbot Sep 13, 2016

Member

#include <cstdint> and use uint64_t.

@klickverbot

klickverbot Sep 13, 2016

Member

#include <cstdint> and use uint64_t.

This comment has been minimized.

@JohanEngelen

JohanEngelen Sep 14, 2016

Member

I had tried it, and it doesn't work :(

@JohanEngelen

JohanEngelen Sep 14, 2016

Member

I had tried it, and it doesn't work :(

This comment has been minimized.

This comment has been minimized.

@JohanEngelen

JohanEngelen Sep 14, 2016

Member

Mac's compiler produces:
pruneCache(char const*, unsigned long, unsigned int, unsigned int, unsigned long long, unsigned int)
DMD and LDC produce:
pruneCache(char const*, unsigned long, unsigned int, unsigned int, unsigned long, unsigned int)

Even the table here https://dlang.org/spec/interfaceToC.html is ambiguous for ulong...

@JohanEngelen

JohanEngelen Sep 14, 2016

Member

Mac's compiler produces:
pruneCache(char const*, unsigned long, unsigned int, unsigned int, unsigned long long, unsigned int)
DMD and LDC produce:
pruneCache(char const*, unsigned long, unsigned int, unsigned int, unsigned long, unsigned int)

Even the table here https://dlang.org/spec/interfaceToC.html is ambiguous for ulong...

This comment has been minimized.

@klickverbot

klickverbot Sep 14, 2016

Member

Sorry, I didn't notice this was actually an extern(C++) interface.

@klickverbot

klickverbot Sep 14, 2016

Member

Sorry, I didn't notice this was actually an extern(C++) interface.

driver/main.cpp
@@ -1171,6 +1172,11 @@ void codegenModules(Modules &modules) {
}
}
+ // Prune the ir2obj cache if needed.
+ if (!opts::ir2objCacheDir.empty()) {

This comment has been minimized.

@klickverbot

klickverbot Sep 13, 2016

Member

Minor comment: I found this a bit misleading to read – especially in conjunction with the comment –, since pruneCache doesn't actually do any pruning unless it is enabled.

@klickverbot

klickverbot Sep 13, 2016

Member

Minor comment: I found this a bit misleading to read – especially in conjunction with the comment –, since pruneCache doesn't actually do any pruning unless it is enabled.

This comment has been minimized.

@JohanEngelen

JohanEngelen Sep 14, 2016

Member

I've moved the check to ir2obj::pruneCache(), and removed the comment.

@JohanEngelen

JohanEngelen Sep 14, 2016

Member

I've moved the check to ir2obj::pruneCache(), and removed the comment.

tests/linking/ir2obj_cache_pruning2.d
+ {
+ // Sleep for 2 seconds, so we are sure that the cache object file timestamps are "aging".
+ import core.thread;
+ Thread.sleep( dur!("seconds")( 2 ) );

This comment has been minimized.

@klickverbot

klickverbot Sep 13, 2016

Member

D style comment: dur!"seconds"(2) would be more in line with upstream/Phobos.

@klickverbot

klickverbot Sep 13, 2016

Member

D style comment: dur!"seconds"(2) would be more in line with upstream/Phobos.

This comment has been minimized.

@JohanEngelen

JohanEngelen Sep 14, 2016

Member

ok, fixed

@JohanEngelen

JohanEngelen Sep 14, 2016

Member

ok, fixed

tests/linking/ir2obj_cache_pruning2.d
+void main()
+{
+ // Add non-zero static data to guarantee a binary size larger than 200000/2.
+ static byte[120000] dummy = 1;

This comment has been minimized.

@klickverbot

klickverbot Sep 13, 2016

Member

D style tip: Underscores (120_000) are great for readability.

@klickverbot

klickverbot Sep 13, 2016

Member

D style tip: Underscores (120_000) are great for readability.

This comment has been minimized.

@JohanEngelen

JohanEngelen Sep 14, 2016

Member

great, fixed

@JohanEngelen

JohanEngelen Sep 14, 2016

Member

great, fixed

@@ -41,8 +42,57 @@
#include "llvm/Support/Path.h"
#include "llvm/Support/raw_ostream.h"
+// Include close() declaration.

This comment has been minimized.

@klickverbot

klickverbot Sep 13, 2016

Member

Pretty sure LLVM guarantees availability of that.

@klickverbot

klickverbot Sep 13, 2016

Member

Pretty sure LLVM guarantees availability of that.

This comment has been minimized.

@JohanEngelen

JohanEngelen Sep 14, 2016

Member

Nope.

This comment has been minimized.

@JohanEngelen

JohanEngelen Sep 14, 2016

Member

Platform support was painful for this PR :( On my Mac, things all worked fine, but to get it to compile and pass tests on all CI systems took a lot of trial and error.

@JohanEngelen

JohanEngelen Sep 14, 2016

Member

Platform support was painful for this PR :( On my Mac, things all worked fine, but to get it to compile and pass tests on all CI systems took a lot of trial and error.

@klickverbot

This comment has been minimized.

Show comment
Hide comment
@klickverbot

klickverbot Sep 13, 2016

Member

I suppose you'll move ldc-profdata over in a separate PR?

Member

klickverbot commented Sep 13, 2016

I suppose you'll move ldc-profdata over in a separate PR?

@JohanEngelen

This comment has been minimized.

Show comment
Hide comment
@JohanEngelen

JohanEngelen Sep 14, 2016

Member

I suppose you'll move ldc-profdata over in a separate PR?

Yep!

Member

JohanEngelen commented Sep 14, 2016

I suppose you'll move ldc-profdata over in a separate PR?

Yep!

@JohanEngelen

This comment has been minimized.

Show comment
Hide comment
@JohanEngelen

JohanEngelen Sep 14, 2016

Member

(CircleCI was green. Currently broken because LLVM 4.0 API change)

Member

JohanEngelen commented Sep 14, 2016

(CircleCI was green. Currently broken because LLVM 4.0 API change)

@klickverbot klickverbot merged commit 1a70704 into ldc-developers:master Sep 14, 2016

2 of 3 checks passed

ci/circleci Your tests failed on CircleCI
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@klickverbot

This comment has been minimized.

Show comment
Hide comment
@klickverbot

klickverbot Sep 14, 2016

Member

We could think about backing out ldc-prune-cache and integrating it into ldc2 before the next release, now that the code is there anyway. Not sure which is the less confusing way.

Member

klickverbot commented Sep 14, 2016

We could think about backing out ldc-prune-cache and integrating it into ldc2 before the next release, now that the code is there anyway. Not sure which is the less confusing way.

@JohanEngelen JohanEngelen deleted the JohanEngelen:prunecache_d branch Sep 15, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment