ci: ccache does not work for macOS cross-compiling builds

[hebasto]

https://cirrus-ci.com/task/6268895175639040?logs=ci#L3869:

ccache version 3.7.7
cache directory                     /tmp/ccache_dir
primary config                      /tmp/ccache_dir/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats updated                       Tue Mar 30 01:48:58 2021
stats zeroed                        Tue Mar 30 01:21:25 2021
cache hit (direct)                     0
cache hit (preprocessed)               0
cache miss                             0
cache hit rate                      0.00 %
called for link                       10
multiple source files                636
cleanups performed                     0
files in cache                         0
cache size                           0.0 kB
max cache size                     200.0 MB

UPD 2023-08-25. See ccache/ccache#1326.

[hebasto]

This line

multiple source files                636

describes a little. According to ccache docs:

The compiler was called to compile multiple source files in one go. This is not supported by ccache.

[hebasto]

@dongcarl

ccache behavior was broken in 3007339 (#19683).

[dongcarl]

Was this not the case before 3007339? I'm not entirely sure how that commit makes it so that "the compiler was called to compile multiple source files in one go"

[hebasto]

I'm not entirely sure how that commit makes it so that "the compiler was called to compile multiple source files in one go"

I believe this is because ccache do not support invoking a compiler that prefixed with env command.

FWIW, if I inject ccache between env and clang, it works fine.

[dongcarl]

FWIW, if I inject ccache between env and clang, it works fine.

What is the ccache stats output if you inject ccache between env and clang?

[hebasto]

@dongcarl

With the following patch

diff --git a/depends/hosts/darwin.mk b/depends/hosts/darwin.mk
index dd71697f0..8a7c3a791 100644
--- a/depends/hosts/darwin.mk
+++ b/depends/hosts/darwin.mk
@@ -98,6 +98,7 @@ $(foreach TOOL,$(cctools_TOOLS),$(eval darwin_$(TOOL) = $$(build_prefix)/bin/$$(
 darwin_CC=env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH \
               -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH \
               -u LIBRARY_PATH \
+            /usr/bin/ccache \
             $(clang_prog) --target=$(host) -mmacosx-version-min=$(OSX_MIN_VERSION) \
               -B$(build_prefix)/bin -mlinker-version=$(LD64_VERSION) \
               --sysroot=$(OSX_SDK) \
@@ -106,6 +107,7 @@ darwin_CC=env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH \
 darwin_CXX=env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH \
                -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH \
                -u LIBRARY_PATH \
+            /usr/bin/ccache \
              $(clangxx_prog) --target=$(host) -mmacosx-version-min=$(OSX_MIN_VERSION) \
                -B$(build_prefix)/bin -mlinker-version=$(LD64_VERSION) \
                --sysroot=$(OSX_SDK) \

• the first run

$ CONFIG_SITE=$PWD/depends/x86_64-apple-darwin18/share/config.site ./configure --disable-ccache
$ ccache --zero-stats
$ make
$ ccache --show-stats 
cache directory                     /home/hebasto/.ccache
primary config                      /home/hebasto/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats updated                       Fri Apr  9 14:03:35 2021
stats zeroed                        Fri Apr  9 13:59:19 2021
cache hit (direct)                   521
cache hit (preprocessed)               1
cache miss                           647
cache hit rate                     44.65 %
called for link                       12
cleanups performed                     2
files in cache                     24070
cache size                          13.0 GB
max cache size                      15.0 GB

• the second run

$ make clean
$ ccache --zero-stats
$ make
$ ccache --show-stats 
cache directory                     /home/hebasto/.ccache
primary config                      /home/hebasto/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats updated                       Fri Apr  9 14:04:47 2021
stats zeroed                        Fri Apr  9 14:04:15 2021
cache hit (direct)                   644
cache hit (preprocessed)               2
cache miss                             1
cache hit rate                     99.85 %
called for link                       11
cleanups performed                     0
files in cache                     24073
cache size                          13.0 GB
max cache size                      15.0 GB

[dongcarl]

Urgh... That kinda sucks... I think the easiest solution is perhaps to wait until ccache 4.1 is available on all distros and set compiler_type/CCACHE_COMPILERTYPE?

[prusnak]

to wait until ccache 4.1 is available on all distros

@dongcarl What do you mean by "all distros"?

[dongcarl]

Sorry, probably better rephrased as "available on stable versions of major distros"... Sometimes I think we might need a configure script for depends lol

[prusnak]

"available on stable versions of major distros"

• Debian 10 (old stable) has ccache 3.6
• Debian 11 (stable) has ccache 4.2
• Ubuntu 20.04 (latest LTS) has ccache 3.7.7
• Ubuntu 21.04 has ccache 4.2
• Ubuntu 21.10 has ccache 4.2.1
• Ubuntu 22.04 (upcoming LTS) has ccache 4.5.1

src: https://repology.org/project/ccache/versions

[hebasto]

@dongcarl

Urgh... That kinda sucks... I think the easiest solution is perhaps to wait until ccache 4.1 is available on all distros and set compiler_type/CCACHE_COMPILERTYPE?

I've tested this approach on Ubuntu 22.04 with ccache 4.5.1, and it doesn't work.

OTOH, another approach has been implemented in #24620.

[Sjors]

Not just when cross-compiling. I made a separate issue #25387, because I'm not sure if it's the same (ccache 4.6.1).

(nvm: it does work as far as I can tell, but doesn't use it by default)

[fanquake]

Given it's not in the description, can someone summarize the current state of this issue?

• What doesn't work?
• What ccache options have been tried (and don't work)? Taking into account recent ccache releases.
• Have we opened an issue upstream to either report that ccache is broken (for us), or that we have a usecase which isn't satified by it's current feature/option set, to see if they are open to adding a new feature/option?
• If there is any (related) upstream discussions/issues, can somebody link to them here?

[hebasto]

Given it's not in the description, can someone summarize the current state of this issue?

• What doesn't work?

• Have we opened an issue upstream to either report that ccache is broken (for us), or that we have a usecase which isn't satified by it's current feature/option set, to see if they are open to adding a new feature/option?

• If there is any (related) upstream discussions/issues, can somebody link to them here?

See ccache/ccache#1326.

[hebasto]

Urgh... That kinda sucks... I think the easiest solution is perhaps to wait until ccache 4.1 is available on all distros and set compiler_type/CCACHE_COMPILERTYPE?

Tested with ccache 4.8.2. It does not work.