ccache fails test suite with clang

[jhpalmieri]

Error:

[ccache-3.2.2] starting testsuite pch
[ccache-3.2.2] SUITE: "pch", TEST: "no -fpch-preprocess, -include" - Expected "cache miss" to be 1, got 0
[ccache-3.2.2] cache directory                     /Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/ccache-3.2.2/src/testdir.49766/.ccache
[ccache-3.2.2] primary config                      /Users/palmieri/Desktop/Sage_stuff/git/sage/local/var/tmp/sage/build/ccache-3.2.2/src/testdir.49766/.ccache/ccache.conf
[ccache-3.2.2] secondary config      (readonly)    /Users/palmieri/Desktop/Sage_stuff/git/sage/local/etc/ccache.conf
[ccache-3.2.2] cache hit (direct)                     0
[ccache-3.2.2] cache hit (preprocessed)               0
[ccache-3.2.2] cache miss                             0
[ccache-3.2.2] preprocessor error                     1
[ccache-3.2.2] files in cache                         0
[ccache-3.2.2] cache size                           0.0 kB
[ccache-3.2.2] max cache size                       4.0 GB
[ccache-3.2.2] TEST FAILED
[ccache-3.2.2] Test data and log file have been left in testdir.49766
[ccache-3.2.2] make[2]: *** [test] Error 1

This can be fixed by upgrading to ccache 3.3.4.

Upstream tarball:
https://www.samba.org/ftp/ccache/ccache-3.3.4.tar.bz2

Component: packages: optional

Author: François Bissey

Branch/Commit: 01c714a

Reviewer: Ralf Stephan

Issue created by migration from https://trac.sagemath.org/ticket/22836

[kiwifb]

comment:1

Is it with both clang and gcc?

[jhpalmieri]

comment:2

Yes, both clang and gcc. If I start with a Sage tarball and I set SAGE_CHECK=yes and SAGE_INSTALL_CCACHE=yes, then I get this failure with gcc. If I use #12426, I get this failure with clang.

On the other hand, if I build Sage first and then do ./sage -f -c ccache, I get this failure with gcc, but with #12426, it skips the tests:

test/main
PASSED: 448 assertions, 88 tests, 10 suites
CC='gcc' ./test.sh
WARNING: Compiler gcc not supported (version: Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1) -- not running tests

[kcrisman]

comment:3

I can confirm both of these (with sage -f -c ccache) on my OSX 10.11 laptop as well.

[kiwifb]

comment:4

OK the message for clang when starting from a sage build is caused by

compiler_version="`$COMPILER --version 2>&1 | head -1`"
case $compiler_version in
    *gcc*|*g++*|2.95*)
        COMPILER_TYPE_GCC=1
        ;;
    *clang*)
        COMPILER_TYPE_CLANG=1
        ;;
    *)
        echo "WARNING: Compiler $COMPILER not supported (version: $compiler_version) -- not running tests" >&2
        exit 0
        ;;
esac

the test is written with the results of a linux install in mind. It would fare better if things were identified from configure using ax_compiler_vendor.m4. It is really curious it fails the way it does if you build it as part of the toolchain.

I think we may have a case where testing on OS X is just not currently supported.

[kiwifb]

comment:5

Hum... So the test suite doesn't pass on OS X with either compilers but for me it is with slightly different errors. I didn't try to do it as part of a sage install. One thing that the test suite doesn't deal with properly is xcode's installed /usr/bin/gcc. It cannot properly identify it as being clang the way it is written. But CC=clang is properly identified (but test suite fails).

The latest version of ccache (3.3.4) still has the detection issue but I haven't checked yet if the test suite passes when configured properly.

[kiwifb]

comment:6

ccache 3.3.4 passes its test suite on OS X sierra with clang and gcc-7.1.0. Gives up gracefully with /usr/bin/gcc

PASSED: 473 assertions, 92 tests, 10 suites
CC='/usr/bin/gcc' ./test.sh
WARNING: Compiler /usr/bin/gcc not supported (version: Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1) -- not running tests

Still to be tested from a spkg but I think we should upgrade if we want the test suite to pass.

[kiwifb]

Author: François Bissey

[kiwifb]

Commit: 01c714a

[kiwifb]

comment:7

This branch upgrade to ccache-3.3.4 which should pass the test. It also probably fixes #24360. I added a patch to support proper detection of /usr/bin/gcc as clang on OS X. That part should be reviewed in combination with #12426.

---

New commits:

01c714a

Upgrade ccache to 3.3.4 and patch to support /usr/bin/gcc on OS X in tests.

[kiwifb]

Branch: u/fbissey/ccache-3.3.4

[jhpalmieri]

comment:8

Can you provide a link to a tarball, please?

[kiwifb]

comment:9

Thanks for reminding me of that important requirement.

[jhpalmieri]

comment:10

On OS X 10.13.2, passes tests with clang but not with Sage's gcc:

Running test suite base......................................................
Running test suite nocpp2......................................................
Running test suite multi_arch.gcc: warning: x86_64 conflicts with i386 (arch flags ignored)
gcc: warning: x86_64 conflicts with i386 (arch flags ignored)
.gcc: warning: x86_64 conflicts with i386 (arch flags ignored)
gcc: warning: x86_64 conflicts with i386 (arch flags ignored)

Skipped test suite serialize_diagnostics [--serialize-diagnostics not supported by compiler]
Running test suite debug_prefix_map.
Running test suite masquerading.
Running test suite hardlink.
Running test suite direct......................................
Running test suite basedir......
Running test suite compression.
Running test suite readonly...
Running test suite readonly_direct..
Running test suite cleanup...........
Running test suite pch.....pch.c:1:17: error: one or more PCH files were found, but they were invalid
 #include "pch.h"
                 ^
pch.c:1:17: error: use -Winvalid-pch for more information
pch.c:1:10: fatal error: pch.h: No such file or directory
 #include "pch.h"
          ^~~~~~~
compilation terminated.
.cc1: error: one or more PCH files were found, but they were invalid
cc1: error: use -Winvalid-pch for more information
cc1: fatal error: pch.h: No such file or directory
compilation terminated.
.cc1: error: one or more PCH files were found, but they were invalid
cc1: error: use -Winvalid-pch for more information
cc1: fatal error: pch.h: No such file or directory
compilation terminated.

FAILED

Test suite:     pch
Test case:      Use .gch, no -fpch-preprocess, -include, sloppiness
Failure reason: Expected "cache miss" to be 1, actual 0

ccache -s:
cache directory                     /Users/palmieri/Desktop/Sage_stuff/sage_builds/VANILLA/sage-8.1/local/var/tmp/sage/build/ccache-3.3.4/src/testdir.10865/.ccache
primary config                      /Users/palmieri/Desktop/Sage_stuff/sage_builds/VANILLA/sage-8.1/local/var/tmp/sage/build/ccache-3.3.4/src/testdir.10865/ccache.conf
secondary config      (readonly)    
cache hit (direct)                     0
cache hit (preprocessed)               0
cache miss                             0
cache hit rate                      0.00 %
preprocessor error                     1
cleanups performed                     0
files in cache                         0
cache size                           0.0 kB
max files                            480

Test data and log file have been left in testdir.10865
make[2]: *** [test] Error 1

[kiwifb]

comment:11

That's new, I hadn't tested sage's gcc on OS X, only a pre-built binary of gcc-7.1.0. I am not submitting my laptop to build gcc again. I'll see if I can reproduce it with a pre made binary of 7.2.0.

When you say it pass with clang, did you set CC=clang or did you let CC default to /usr/bin/gcc? The first case is no problem, the other is why I added a patch.

[jhpalmieri]

comment:12

Replying to @kiwifb:

When you say it pass with clang, did you set CC=clang or did you let CC default to /usr/bin/gcc? The first case is no problem, the other is why I added a patch.

I just built with this plus #12426, so I let CC default to /usr/bin/gcc.

[kiwifb]

comment:13

Replying to @jhpalmieri:

Replying to @kiwifb:

When you say it pass with clang, did you set CC=clang or did you let CC default to /usr/bin/gcc? The first case is no problem, the other is why I added a patch.

I just built with this plus #12426, so I let CC default to /usr/bin/gcc.

OK, that's one positive out of that. I'll have to figure out what is happening with gcc from sage. Any chance of you checking things on linux to see if it is OS X only?

[jhpalmieri]

comment:14

I don't have easy access to any linux machine, unfortunately, nor do I have a virtual linux box on this one.

[kiwifb]

comment:15

That's OK I have linux machines, I should eat my own dog food a little bit.

[kiwifb]

comment:16

Replying to @kiwifb:

That's OK I have linux machines, I should eat my own dog food a little bit.

All right passes its tests here with sage's own gcc on Gentoo linux. So it is a OS X only issue.

I note that it looks like the original report. I haven't tested sage's own gcc during this ticket because I don't like building it on my laptop. But pre-built gcc binaries have been fine. So I am starting to think that the root cause of this particular report is the way sage builds gcc on OS X.

[kiwifb]

comment:17

Now that building sage on OS X uses clang by default, do we care so much about the problem with sage's own gcc?

[rwst]

comment:18

Replying to @kcrisman:

I can confirm both of these (with sage -f -c ccache) on my OSX 10.11 laptop as well.

Note that sage -f -c ccache does not start the testsuite here (bug). I could only force that with SAGE_CHECK='yes' ./sage -f ccache. I confirm that 3.3.4 passes the testsuite with clang-5.0.1 on Linux.

[rwst]

Reviewer: Ralf Stephan

[rwst]

comment:19

LGTM. Sage builds and, after I touched the Cython files under libs/, the recompile was obviously cached. There is no point in running the testsuite.

[vbraun]

Changed branch from u/fbissey/ccache-3.3.4 to 01c714a