ARROW-2238: [C++] Detect and use clcache in cmake configuration

[pitrou]

No description provided.

[pitrou]

Example AppVeyor build at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.155

[pitrou]

The failure at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.155/job/q31movster4v84d9 shows this can lead to inconsistencies or errors: cmake first tries to detect the compller from user-supplied information (generator, environment variables), then the clcache setting overrides that detection.

Either we add logic to try and avoid such errors, or we simply let people override CC/CXX if they want to use clcache (statu quo).

[pitrou]

Also I'm not sure whether we have a Windows developer on board; I'm merely launching a VM from time to time but otherwise work on Ubuntu :-)

[wesm]

@Maxris can take a look. How does the error you linked to arise?

[pitrou]

I think that's because CMAKE_CXX_COMPILER forcefully overrides the compiler command. When using the Visual Studio generators, you traditionally don't need to run vcvarsall.bat (presumably because cmake would hardcode the full compiler path), but then clcache fails finding the compiler.

So it's possible that calling vcvarsall.bat is all that's needed here. But that would also change the workflow people may be accustomed to.

[maxhora]

@pitrou I will check, thanks

[maxhora]

@pitrou it seems that we already try to use ccache there if it's presented. I'm wondering if it will make more sense to refactor referenced lines and optionally use clcache for MSVC?
Also, probably, usage of RULE_LAUNCH_COMPILE and RULE_LAUNCH_LINK should solve issue with selected compiler overwrite.
And it seems that starting from Cmake 3.4.0 CXX_COMPILER_LAUNCHER variable is available, but we stick to CMake of min ver 3.2

[pitrou]

Also, probably, usage of RULE_LAUNCH_COMPILE and RULE_LAUNCH_LINK should solve issue with selected compiler overwrite.

Last I tried it seemed it didn't work. I might give it a try again...

[maxhora]

I will try on my end as well

[maxhora]

@pitrou, do you have an idea how to verify that clcache.exe was really used during compilation? I've tried with it and without, but I can't find any difference in output/produced results.

[pitrou]

@Maxris running clcache -s gives you aggregate statistics for the cache, so you can see (by the number of hits and misses) if clcache was used at all.

[maxhora]

@pitrou thanks!

[maxhora]

update: it seems that current solution set(CMAKE_CXX_COMPILER ${CLCACHE_FOUND}) works only with NMake Makefiles generator, but clcache doesn't get called if Visual Studio 14 2015 Win64 or similar is used.

[pitrou]

clcache works best with Ninja or NMake (*). My suggestion here would be to recommend Ninja + clcache for best build performance. The other concern, though, is to avoid breaking existing builds for those who prefer other generators (e.g. Visual Studio).

(*) See the following links:

• Enable clcache in CMakeLists.txt frerich/clcache#273 (comment)
• https://github.com/frerich/clcache/wiki/Integration#integration-for-visual-studio

[maxhora]

@pitrou it should be fine to set set(CMAKE_CXX_COMPILER ${CLCACHE_FOUND}) only if Generator defined as Ninja or NMake Makefiles. This possibly also will resolve current Appveyor failure. What do you think?

[pitrou]

@Maxris that sounds ok to me.

[maxhora]

@pitrou I will check if that resolves Appveyor build failure and let you know, thanks

[maxhora]

@pitrou here it's changes to solve Jenkins failure as discussed
Passed Appveyor build

P.S. I've tried to create PR into your pitrou:ARROW-2238-cmake-clcache remote branch, but for some reasons it's not listed in targets during PR creation.

[pitrou]

@Maxris thanks! I'll integrate it in my PR.

[pitrou]

Actually, it doesn't look right. Here you see that the cache wasn't looked up at all:
https://ci.appveyor.com/project/MaxRisuhin/arrow/build/1.0.253/job/puv48c73yknmel34#L3175

[maxhora]

@pitrou sorry, haven't checked in such manner. I have checked only that clcache was applied as compiler replacement with additional output https://ci.appveyor.com/project/MaxRisuhin/arrow/build/1.0.253/job/e8v49o78ku8cltur#L418 ... and also have seen that with NMake Makefiles generator clcache is used in local builds. Will try to figure out why it doesn't work in Appveyor env.

[maxhora]

Hey @pitrou , could you please check logs of last passed build ( https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/build/1.0.5601 ). For some reasons clcache statistics shows that it was used, by my opinion, fine. Could you please verify? Thanks
P.S. It also should be normal that JOB=Cmake_Script_Tests doesn't generate cache entries.

[pitrou]

@Maxris, yes, it looks fine... And the build times are ok too (without cache, the total duration would probably be much more than 45 minutes).

[maxhora]

Great! Should be fine to go with these changes, thanks!

[xhochy]

@Maxris @pitrou This also looks fine from my side but as you both are more qualified on that matter: Is this ready to be merged?

[pitrou]

@xhochy, yes, it's ready for merging.

[wesm]

+1