Compiler dependency information incorrectly shared between builds

[mikecrowe]

We've just upgraded to ccache-3.3.3 and are suffering from problems when compiling kernels under the OpenEmbedded build framework.

The fundamental problem is that dependencies retrieved from the ccache for scripts/genksyms/parse.tab.c contain paths that no longer exist. This upsets the kernel fixdeps tool.

I've attached a script that can be used to reproduce the problem. I used a v4.9 kernel, but it should happen with any even slightly recent kernel version. Uncommenting the CCACHE_DISABLE line causes it to run correctly. Leaving ccache enabled results in:

HOSTCC scripts/genksyms/parse.tab.o
fixdep: error opening config file: ../kernel2/scripts/genksyms/genksyms.h: No such file or directory
scripts/Makefile.host:124: recipe for target 'scripts/genksyms/parse.tab.o' failed

The generated build3/scripts/genksyms/.parse.tab.o.d file contains:

../kernel2/scripts/genksyms/genksyms.h

It seems that during the first build there are no matches in the cache, so both parse.tab.o and .parse.tab.o.d are stored in the cache.

During the second build, the preprocessed output matches, so parse.tab.o is retrieved from the ccache but .parse.tab.o.d is written back to the cache.

During the third build (back in the original source directory), the direct hash matches so both parse.tab.o and .parse.tab.o.d are retrieved from the cache. Unfortunately .parse.tab.o.d refers to the wrong source tree.

Or, from the log file:

[2017-02-15T14:55:58.923445 28698] Stored in cache: scripts/genksyms/parse.tab.o -> /overflow/mac/nobackup/temp/ccache/1/7/845549b05c6daa4ccc906704db3091-924114.o
[2017-02-15T14:55:58.923534 28698] Stored in cache: scripts/genksyms/.parse.tab.o.d -> /overflow/mac/nobackup/temp/ccache/1/7/845549b05c6daa4ccc906704db3091-924114.d
[2017-02-15T14:56:08.122375 9805 ] Created from cache: /overflow/mac/nobackup/temp/ccache/1/7/845549b05c6daa4ccc906704db3091-924114.o -> scripts/genksyms/parse.tab.o
[2017-02-15T14:56:08.162218 9805 ] Stored in cache: scripts/genksyms/.parse.tab.o.d -> /overflow/mac/nobackup/temp/ccache/1/7/845549b05c6daa4ccc906704db3091-924114.d
[2017-02-15T14:56:16.101702 23540] Created from cache: /overflow/mac/nobackup/temp/ccache/1/7/845549b05c6daa4ccc906704db3091-924114.o -> scripts/genksyms/parse.tab.o
[2017-02-15T14:56:16.102046 23540] Created from cache: /overflow/mac/nobackup/temp/ccache/1/7/845549b05c6daa4ccc906704db3091-924114.d -> scripts/genksyms/.parse.tab.o.d

The original dependencies file emitted by the compiler contains:

../temp/kernel1/scripts/genksyms/genksyms.h

(The cmd_and_fixdep definition in scripts/Kbuild.include can be tweaked not to delete $(depfile) so this gets left behind.)

With ccache-3.1.12 and 3.2.3 the two source trees seem to get different hashes so the failure does not occur:

[2017-02-15T15:25:07.457857 22409] Copying scripts/genksyms/parse.tab.o to /overflow/mac/nobackup/temp-old/ccache/e/b/86af9bc852ed06c5668fb67de8d374-932536.o via /overflow/mac/nobackup/temp-old/ccache/e/b/86af9bc852ed06c5668fb67de8d374-932536.o.chuckie.22409.oJavFr (uncompressed)
[2017-02-15T15:25:07.457920 22409] Copying scripts/genksyms/.parse.tab.o.d to /overflow/mac/nobackup/temp-old/ccache/e/b/86af9bc852ed06c5668fb67de8d374-932536.d via /overflow/mac/nobackup/temp-old/ccache/e/b/86af9bc852ed06c5668fb67de8d374-932536.d.chuckie.22409.JZ9AHr (uncompressed)
[2017-02-15T15:25:16.843599 4550 ] Copying scripts/genksyms/parse.tab.o to /overflow/mac/nobackup/temp-old/ccache/f/8/d71f821e239cb4f43bc41c73bbc892-932536.o via /overflow/mac/nobackup/temp-old/ccache/f/8/d71f821e239cb4f43bc41c73bbc892-932536.o.chuckie.4550.GsJNAi (uncompressed)
[2017-02-15T15:25:16.843664 4550 ] Copying scripts/genksyms/.parse.tab.o.d to /overflow/mac/nobackup/temp-old/ccache/f/8/d71f821e239cb4f43bc41c73bbc892-932536.d via /overflow/mac/nobackup/temp-old/ccache/f/8/d71f821e239cb4f43bc41c73bbc892-932536.d.chuckie.4550.R3n5tS (uncompressed)
[2017-02-15T15:25:23.827790 18604] Copying /overflow/mac/nobackup/temp-old/ccache/e/b/86af9bc852ed06c5668fb67de8d374-932536.o to scripts/genksyms/parse.tab.o via scripts/genksyms/parse.tab.o.chuckie.18604.LOgsz9 (uncompressed)
[2017-02-15T15:25:23.827846 18604] Created from cache: /overflow/mac/nobackup/temp-old/ccache/e/b/86af9bc852ed06c5668fb67de8d374-932536.o -> scripts/genksyms/parse.tab.o
[2017-02-15T15:25:23.827860 18604] Copying /overflow/mac/nobackup/temp-old/ccache/e/b/86af9bc852ed06c5668fb67de8d374-932536.d to scripts/genksyms/.parse.tab.o.d via scripts/genksyms/.parse.tab.o.d.chuckie.18604.HwP489 (uncompressed)
[2017-02-15T15:25:23.827882 18604] Created from cache: /overflow/mac/nobackup/temp-old/ccache/e/b/86af9bc852ed06c5668fb67de8d374-932536.d -> scripts/genksyms/.parse.tab.o.d

It seems that ccache can't consider the hash of the preprocessed output to be sufficient when the dependencies file is being generated. Perhaps the -I options need to be thrown into the hash when -Wp,-MD is passed to the compiler?

old.log.gz
new.log.gz
repro.sh

[jrosdahl]

Thanks for the excellent report and reproduction recipe!

This bug has been fixed in 7cad4ac; see #134. Not yet released though, but I will make a 3.3.4 release soon.

[mikecrowe]

Thanks for the quick response! I'd re-used an old tree that was still pulling from git.samba.org so I didn't spot the fix when I went searching in the history for likely candidates. :(