ccache support #3761

junghans · 2017-04-07T16:22:20Z

export SPACK_ENABLE_CCACHE=yes
export PATH="/path/to/cache${PATH:+:}${PATH}"
spack install libelf
ccache -z
spack uninstall libelf && spack install libelf
ccache -s

and yeah

cache directory                     /home/junghans/.ccache
primary config                      /home/junghans/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
cache hit (direct)                   114
cache hit (preprocessed)               0
cache miss                             4
called for link                        1
called for preprocessing              18
compile failed                         2
autoconf compile/link                 21
files in cache                       394
cache size                           5.8 MB
max cache size                       5.0 GB

adamjstewart · 2017-04-07T16:24:13Z

It might be better to provide this support through Spack's configuration file: ~/.spack/configuration.yaml. I don't like relying on environment variables. I don't actually know how to add entries to configuration.yaml or use them in Spack, but @tgamblin should be able to help.

junghans · 2017-04-07T16:32:09Z

It is a start! If you tell me how to query Spack's configuration in cc, I will do that.
Plus, it would be nice to point it at a ccache binary installed by spack itself instead of relying on $PATH.

junghans · 2017-04-07T16:59:51Z

On a related note, for clang we should export CCACHE_CPP2=yes to avoid endless warnings like:

clang: warning: argument unused during compilation: '-c'
clang: warning: argument unused during compilation: '-I .'

(Details on why here: http://peter.eisentraut.org/blog/2014/12/01/ccache-and-clang-part-3/)

citibeth · 2017-04-08T02:54:43Z

What is the expected use case here?

junghans · 2017-04-09T23:07:42Z

@citibeth : I like the idea to give the users an easy way to enable compiler caching.

citibeth · 2017-04-10T03:34:17Z

Yes, I understand that. But I don't understand why it would be helpful. Compiler caching is useful when you compile the same thing over and over again --- as when you are engaged in the edit/compile/run debug cycle. Most of what Spack does is compile things once.

…

On Sun, Apr 9, 2017 at 7:07 PM, Christoph Junghans ***@***.*** > wrote: @citibeth <https://github.com/citibeth> : I like the idea to give the users an easy way to enable compiler caching. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3761 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB1cdydLth_F7BM-EeM6q7F8F-WwurBkks5ruWTAgaJpZM4M3Guh> .

davydden · 2017-04-10T05:41:24Z

@junghans:

My understanding is similar to @citibeth 's in that we won't get anything from ccache in Spack unless it's integrated with spack setup to help developers use Spack's machinery to setup/config their package and get all benefits of caching when re-building their library while developing.

If that is not the case, maybe you could illustrate the difference in compile time on some big package like trilinos?

I also agree with @adamjstewart that such things should be configured through yaml files.

junghans · 2017-04-10T15:37:13Z

@citibeth: There are hundreds of blog posts out there showing that ccache is or isn't useful for a particular package, however, from my own experience (on Gentoo) ccache comes in handy if you need to compile similar variants of the same package (same compiler, but different features or different patch sets). Also the time saving depends a lot on the machine and the filesystem, where the cache is store, it is the usual compute vs. I/O trade-off.

@davydden: The libelf example above shows that spack can useccache (cache hit (direct) 114) by simply prefixing the compiler with the ccache binary. Sure, as discussed it would be nicer to integrate this feature using spack setup instead of an environment variable, but let's first decide if we want to support compiler caching.

citibeth · 2017-04-10T16:19:18Z

Thanks for the the use case; this should go in the docs when you write it up. Specifically, two use cases: (1) Use with `spack setup`, (2) Build the same package with many different variants. Should this PR also try to integrate f90cache? https://perso.univ-rennes1.fr/edouard.canot/f90cache/

…

On Mon, Apr 10, 2017 at 11:37 AM, Christoph Junghans < ***@***.***> wrote: @citibeth <https://github.com/citibeth>: There are hundreds of blog posts out there showing that ccache is or isn't useful for a particular package, however, from my own experience (on Gentoo) ccache comes in handy if you need to compile similar variants of the same package (same compiler, but different features or different patch sets). Also the time saving depends a lot on the machine and the filesystem, where the cache is store, it is the usual compute vs. I/O trade-off. @davydden <https://github.com/davydden>: The libelf example above shows that spack can useccache (cache hit (direct) 114) by simply prefixing the compiler with the ccache binary. Sure, as discussed it would be nicer to integrate this feature using spack setup instead of an environment variable, but let's first decide if we want to support compiler caching. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3761 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB1cd9MppyJQVuEuG07pvH7vftfJ71STks5rukyqgaJpZM4M3Guh> .

junghans · 2017-04-10T17:16:09Z

lib/spack/spack/build_environment.py

@@ -336,6 +337,10 @@ def set_build_environment_variables(pkg, env, dirty=False):
    env.set(SPACK_SHORT_SPEC, pkg.spec.short_spec)
    env.set(SPACK_DEBUG_LOG_DIR, spack.spack_working_dir)

+    # Find ccache binary and hand it to build environment
+    if spack.ccache:
+        env.set(SPACK_CCACHE_BINARY, 'ccache')


Here is the crux, how would I detect a ccache binary built by spack previously.

junghans · 2017-04-10T17:19:09Z

@citibeth: see my latest commit.

Should this PR also try to integrate f90cache?

I will do that in another pull request.

citibeth

Good start, more complete docs would be great!

citibeth · 2017-04-10T19:15:09Z

lib/spack/docs/config_yaml.rst

+When set to ``true`` Spack will use ccache to cache compiles. This is
+useful specifically un two cases: (1) Use with ``spack setup``, (2)
+Build the same package with many different variants. The default is
+``false``.


Could more documentation be useful? For example... How does Spack find ccache? Are you supposed to ask Spack to build it first? How should ccache be configured?

How does Spack find ccache?

You tell me, how you want it (see my comment above). Currently spack just assumes ccache is in PATH. And I can certainly write that here.

Are you supposed to ask Spack to build it first?

Sure, if you don't have it on your system that is an option.

How should ccache be configured?

ccache works nicely out of the box, but we can of course refer the user to the Configuration settings of man ccache.

adamjstewart · 2017-04-10T20:16:43Z

lib/spack/docs/config_yaml.rst

+ccache`. ``ccache`` comes with reasonable defaults for cache size
+and location. (See the *Configuration settings* secion of `man
+ccache` to learn more about the default settings and how change
+them.)


spack install ccache and man ccache need double backticks around them.

adamjstewart · 2017-04-10T20:17:04Z

lib/spack/docs/config_yaml.rst

+--------------------
+
+When set to ``true`` Spack will use ccache to cache compiles. This is
+useful specifically un two cases: (1) Use with `spack setup`, (2)


spack setup needs double backticks.

junghans · 2017-04-11T20:26:01Z

@citibeth: Any further comments?

citibeth · 2017-04-11T21:56:28Z

lib/spack/env/cc

@@ -339,7 +339,11 @@ case "$mode" in
        args=("${args[@]}" ${SPACK_LDLIBS[@]}) ;;
 esac

-full_command=("$command" "${args[@]}")
+if [[ ${lang_flags} != "F" && ${SPACK_CCACHE_BINARY} ]]; then


Add a comment explaining what ${lang_flags}!="F" means. I think you're filtering out the Fortran compiler.

junghans · 2017-04-12T16:47:01Z

So, can somebody test this please, e.g.:

git fetch origin pull/3761/head && git checkout FETCH_HEAD
spack install ccache
sed -i '/ccache:/s/false/true/' config.yaml
spack install libelf
ccache -z
spack uninstall libelf && spack install libelf
ccache -s

healther · 2017-04-13T09:45:29Z

$ ccache -z
$ spack uninstall libelf && spack install libelf
$ ccache -s
cache directory                     ~/.ccache
primary config                      ~/.ccache/ccache.conf
secondary config      (readonly)   ~/Documents/git/spack/opt/spack/1-debian8-x86_64/gcc-4.9.2/ccache-3.3.4-n5cd2eq5a47jcidgkq7ffejlp3g7wuhx/etc/ccache.conf
cache hit (direct)                     0
cache hit (preprocessed)               0
cache miss                           118
cache hit rate                      0.00 %
called for link                        1
called for preprocessing              18
compile failed                         2
autoconf compile/link                 19
cleanups performed                     0
files in cache                       415
cache size                           5.6 MB
max cache size                       5.0 GB

junghans · 2017-04-13T13:46:51Z

@healther : Hmm, strange.

files in cache 415

The good news is that ccache was used, so the code in this PR works.

cache miss 118

The bad news is, there were no hits, so for some reason ccache thought it couldn't reuse cache.

junghans · 2017-04-13T14:04:14Z

@healther: Which compiler were you using?

healther · 2017-04-13T14:05:37Z

The system-provided compiler

gcc --version
gcc (Debian 4.9.2-10) 4.9.2
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

healther · 2017-04-13T14:06:33Z

The number of files in the cache increases each time I uninstall and reinstall

healther · 2017-04-13T14:44:58Z

FYI: on my private machine (MacOS, with python3) I get the same behaviour:

$ ccache -s
cache directory                     /Users/healther/.ccache
primary config                      /Users/healther/.ccache/ccache.conf
secondary config      (readonly)    /Users/healther/Documents/git/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/ccache-3.3.3-3h5lzitlf55lxrzq7vnyuoy5qq7bgno4/etc/ccache.conf
cache hit (direct)                     0
cache hit (preprocessed)               0
cache miss                           112
cache hit rate                      0.00 %
called for preprocessing              36
compile failed                         2
autoconf compile/link                 36
cleanups performed                     0
files in cache                       220
cache size                           2.4 MB
max cache size                       5.0 GB

junghans · 2017-11-28T23:42:14Z

@scheibelp SC17 is over, so time to merge!

healther · 2018-01-05T17:15:45Z

Could we please merge this? We would like to start autodeploying software with spack builds, and as long as spack is not stackable (using one spack's packages as external ones for another instance) ccache would at least somewhat speedup our build process.

JavierCVilla · 2018-01-18T09:37:49Z

Could we please merge this? We would like to start autodeploying software with spack builds, and as long as spack is not stackable

Same case here, it would be nice to have such a feature.

healther · 2018-01-22T14:24:17Z

@junghans do you have any known-issues or similar that I should be aware of when using this PR?

junghans · 2018-01-22T16:32:18Z

@healther: it should just work ;-) Can you give it another test run? So that @tgamblin or @scheibelp can finally merge this.

healther · 2018-01-23T16:20:14Z

I'll take a look and report back :)

healther · 2018-01-24T12:35:02Z

Mh, I see a massive slow down of the configure step.
without ccache

==> Successfully installed perl
Fetch: 0.94s.  Build: 6m 12.32s.  Total: 6m 13.26s.

[+] /wang/users/wv385/cluster_home/tmp/20180122_staging/opt/spack/linux-debian8-x86_64/gcc-4.9.2/perl-5.24.1-xtnzd3og6g3kbzvqhwhqqdyswcepnhap

with ccache the

==> Building perl [Package]
==> Executing phase: 'configure'

takes significantly longer than that. Now that might be a problem of our nfs filesystem mount (which currently has some timeout problems). I'll test it also on a scratch filesystem (i.e. locally mounted, not sure about the terminology)
Here I see a difference of (with ccache)

==> Successfully installed perl 
  Fetch: 0.03s.  Build: 2m 34.52s.  Total: 2m 34.55s.
[+] /tmp/blub1/20180122_staging/opt/spack/linux-debian8-x86_64/gcc-4.9.2/perl-5.24.1-xtnzd3og6g3kbzvqhwhqqdyswcepnhap

without ccache

==> Successfully installed perl
  Fetch: 0.51s.  Build: 1m 36.99s.  Total: 1m 37.50s. 
[+] /tmp/blub1/20180122_staging/opt/spack/linux-debian8-x86_64/gcc-4.9.2/perl-5.24.1-xtnzd3og6g3kbzvqhwhqqdyswcepnhap

and the with ccache is with a warm cache (i.e. perl was already build with it once, albeit in a different folder) and I see a lot of hits in the statistics!

Any ideas what is happening here @junghans ?

healther · 2018-01-24T12:35:59Z

Can I get timings for the single steps from spack? I'd expect that build is actually faster and it's only configure that is slow

healther · 2018-01-29T08:54:34Z

Okay @junghans current status:

Good news: The weird build-time increase was due to our nfs-filesystem, changing the cache location seems to remove the penalty
Bad news: Even though I see some cachehits (O(20%)), the build-time does not significantly change (even after spack install && spack uninstall && spack install). Do I need to set additional environment variables besides CCACHE_DIR?

junghans · 2018-01-30T01:50:03Z

It really depends on how reproducible the build is, __DATE__ and friends are bad!

junghans · 2018-05-03T17:44:51Z

Ping @tgamblin

paul-chelarescu · 2018-06-28T13:09:31Z

This feature is working for my case as well, and giving significant built time improvements. Could this be merged soon?

junghans · 2018-06-28T23:55:32Z

Ping @tgamblin @scheibelp

adamjstewart · 2018-06-29T01:14:56Z

It looks like the unit tests are failing.

scheibelp · 2018-06-29T01:33:20Z

lib/spack/spack/build_environment.py

@@ -335,6 +336,13 @@ def set_build_environment_variables(pkg, env, dirty):
    env.set(SPACK_DEBUG_LOG_ID, pkg.spec.format('${PACKAGE}-${HASH:7}'))
    env.set(SPACK_DEBUG_LOG_DIR, spack.main.spack_working_dir)

+    # Find ccache binary and hand it to build environment
+    if spack.config.get('ccache'):


I think this is causing the latest failing tests - it should be something like if spack.config.get('config:ccache'):

scheibelp

Currently the tests are still failing based on retrieving the value of ccache from config.yaml (you are using a . instead of a : to specify a property of a section)

I also have a minor request for deciding when to enable ccache support in the cc wrapper.

Other than that this looks good (which is to say I'm of a mind to finally merge it). I won't get a chance to merge this until 7/9 though.

scheibelp · 2018-06-29T21:58:29Z

lib/spack/spack/build_environment.py

@@ -335,6 +336,13 @@ def set_build_environment_variables(pkg, env, dirty):
    env.set(SPACK_DEBUG_LOG_ID, pkg.spec.format('${PACKAGE}-${HASH:7}'))
    env.set(SPACK_DEBUG_LOG_DIR, spack.main.spack_working_dir)

+    # Find ccache binary and hand it to build environment
+    if spack.config.get('config.ccache'):


This should be config:ccache vs config.ccache (need a colon vs. a period)

scheibelp · 2018-06-29T22:00:52Z

lib/spack/env/cc

@@ -342,7 +342,15 @@ case "$mode" in
        args=("${args[@]}" ${SPACK_LDLIBS[@]}) ;;
 esac

-full_command=("$command" "${args[@]}")
+#ccache only supports C languages, so filtering out Fortran
+if [[ ${lang_flags} != "F" && ${SPACK_CCACHE_BINARY} ]]; then


Could we whitelist vs. blacklist here? e.g. if ${lang_flags} = "C" || ${lang_flags} = "CXX" (i probably didn't get that syntax right).

I'm aware that C/CXX/F are the only possible values currently but it's generally less fragile to specify the cases where it should enable ccache.

scheibelp · 2018-07-09T21:06:17Z

Thanks!

If the user sets "ccache: true" in spack's config.yaml, Spack will use an available ccache executable when compiling c/c++ code. This feature is disabled by default (i.e. "ccache: false") and the documentation is updated with how to enable ccache support

junghans commented Apr 10, 2017

View reviewed changes

adamjstewart approved these changes Apr 10, 2017

View reviewed changes

citibeth suggested changes Apr 10, 2017

View reviewed changes

adamjstewart reviewed Apr 10, 2017

View reviewed changes

junghans force-pushed the ccache branch from 89aac08 to af34546 Compare April 10, 2017 20:18

junghans mentioned this pull request Apr 11, 2017

Update ccache version to 3.3.4 #3794

Merged

junghans force-pushed the ccache branch from af34546 to 6b41b28 Compare April 11, 2017 20:33

citibeth approved these changes Apr 11, 2017

View reviewed changes

junghans force-pushed the ccache branch 2 times, most recently from 188b30f to 6d73800 Compare April 12, 2017 02:25

junghans changed the title ~~Rudimentary ccache support~~ ccache support Apr 12, 2017

junghans force-pushed the ccache branch 2 times, most recently from e0cea99 to 4ae964e Compare November 28, 2017 17:47

junghans assigned scheibelp Jan 3, 2018

alalazo mentioned this pull request Jan 15, 2018

Can we reduce Travis build times #6928

Closed

scheibelp reviewed Jun 29, 2018

View reviewed changes

junghans force-pushed the ccache branch from 02c5336 to 7b85454 Compare June 29, 2018 04:04

scheibelp requested changes Jun 29, 2018

View reviewed changes

added ccache support

7b7d05c

junghans force-pushed the ccache branch from 7b85454 to 7b7d05c Compare June 29, 2018 23:30

scheibelp approved these changes Jul 9, 2018

View reviewed changes

scheibelp merged commit 8bc3f7d into spack:develop Jul 9, 2018

junghans deleted the ccache branch July 10, 2018 23:14

This was referenced Jul 30, 2018

Use a bot to mark stale issues #8811

Closed

Enable ccache in travis #6370

Merged

ccache support #3761

ccache support #3761

Conversation

junghans commented Apr 7, 2017 • edited Loading

adamjstewart commented Apr 7, 2017 • edited by tgamblin Loading

junghans commented Apr 7, 2017

junghans commented Apr 7, 2017

citibeth commented Apr 8, 2017

junghans commented Apr 9, 2017

citibeth commented Apr 10, 2017 via email

davydden commented Apr 10, 2017

junghans commented Apr 10, 2017

citibeth commented Apr 10, 2017 via email

Choose a reason for hiding this comment

junghans commented Apr 10, 2017

citibeth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

junghans commented Apr 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

junghans commented Apr 12, 2017

healther commented Apr 13, 2017 • edited Loading

junghans commented Apr 13, 2017

junghans commented Apr 13, 2017

healther commented Apr 13, 2017

healther commented Apr 13, 2017

healther commented Apr 13, 2017

junghans commented Nov 28, 2017

healther commented Jan 5, 2018

JavierCVilla commented Jan 18, 2018

healther commented Jan 22, 2018

junghans commented Jan 22, 2018

healther commented Jan 23, 2018

healther commented Jan 24, 2018

healther commented Jan 24, 2018

healther commented Jan 29, 2018

junghans commented Jan 30, 2018

junghans commented May 3, 2018

paul-chelarescu commented Jun 28, 2018

junghans commented Jun 28, 2018

adamjstewart commented Jun 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scheibelp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scheibelp commented Jul 9, 2018

junghans commented Apr 7, 2017 •

edited

Loading

adamjstewart commented Apr 7, 2017 •

edited by tgamblin

Loading

healther commented Apr 13, 2017 •

edited

Loading