Hard link mode can corrupt .d files in the cache #599
Comments
|
Thanks for the bug report. I can't reproduce the bug from the description, though. It doesn't seem likely that such a standard case would be badly handled by ccache (since somebody would have reported it already), so my guess is that there has to be something more subtle in play. But maybe the use of the depend mode makes it subtle or at least an uncommon enough setup.
The log line you pointed out does not by itself tell much about if it's a bug or not. It says that a header file was not found for one of the result entries when looking for a match in the manifest file, but a manifest file can store several result entries and the subsequent log line ( It's of interest to take a look at the manifest file. In this case For debugging purposes the ideal set of information to work with would be:
For reference, here is a script I wrote when trying to reproduce the bug with ccache 3.7.9: #!/bin/sh
set -eu
tmpdir=$(mktemp -d -p .)
trap "cd /; rm -rf $tmpdir" EXIT
cd $tmpdir
export CCACHE_DEBUG=1
export CCACHE_DEPEND=1
export CCACHE_DIR=ccache.dir
export CCACHE_HARDLINK=1
export CCACHE_SLOPPINESS=include_file_mtime,include_file_ctime
ccache=${CCACHE:-ccache}
cc=${CC:-cc}
(echo '#include "a.hpp"'; echo 'int main(){return 0;}') >a.cpp
echo '#include "b.hpp"' >a.hpp
echo '// hi there!' >b.hpp
$ccache $cc -c -o a.o -MD -MF a.m a.cpp
grep Result a.o.ccache-log
cat a.m
echo
rm b.hpp
echo >a.hpp
$ccache $cc -c -o a.o -MD -MF a.m a.cpp
grep Result a.o.ccache-log
cat a.mOutput for me: |
|
Thanks for the fast response. I agree that it seems unlinkely that such a bug goes unnoticed for any extended amount of time, yet I was able to reproduce the result I consider bogus by simply removing the three lines dealing with the tmpdir in the reproducer you provided (thanks for that, I should really have done this). I see expected output for the first run: But for the second run, without touching anything: Note that the first |
|
Thanks. Now I understand. The issue is not the depend mode, it's the hard link mode. I would guess that the hard link mode is rarely used, which explains why the issue hasn't been reported before. What happens is this:
Thus the second compilation corrupts cache entry 1. This is an instance of the warning present for the
The compiler always unlinks the object file before writing a new one, but it doesn't do that for the dependency file, which is news to me. So the quick fix is to not use the hard link mode since it's dangerous. A slow fix is to wait for ccache 4.0 which won't store dependency files as hard links and also will detect size changes of hard-linked object files (a46fd5b). Edit: I will consider making 3.7.10 never use hard links for the |
The compiler unlinks the destination object file before writing, but it apparently doesn’t do that for dependency files. This means that compilation can corrupt a .d file that shares i-node with a cached .d file when using the hard link mode. Here is a scenario where this can happen: 1. There is a test.c which includes test.h. 2. When test.c is compiled, the compiler writes test.d which mentions test.h and ccache hard links test.d into cache entry 1. test.d and cache entry 1's .d file now share i-nodes. 3. The include of test.h is removed from test.c. 4. When test.c is compiled again the compiler overwrites test.d with new content without test.h and ccache hard links test.d into cache entry 2. test.d, cache entry 1 and cache entry 2 now share i-nodes, all of which contain the new content without test.h. Since we can’t be sure how the compiler behaves for other types of files (.dwo, .cov, etc.), only try to to hard link object files. Fixes ccache#599.
The compiler unlinks the destination object file before writing, but it apparently doesn’t do that for dependency files. This means that compilation can corrupt a .d file that shares i-node with a cached .d file when using the hard link mode. Here is a scenario where this can happen: 1. There is a test.c which includes test.h. 2. When test.c is compiled, the compiler writes test.d which mentions test.h and ccache hard links test.d into cache entry 1. test.d and cache entry 1's .d file now share i-nodes. 3. The include of test.h is removed from test.c. 4. When test.c is compiled again the compiler overwrites test.d with new content without test.h and ccache hard links test.d into cache entry 2. test.d, cache entry 1 and cache entry 2 now share i-nodes, all of which contain the new content without test.h. Since we can’t be sure how the compiler behaves for other types of files (.dwo, .cov, etc.), only try to to hard link object files. Fixes ccache#599. (cherry picked from commit 443afc1)
|
Fixed on 3.7-maint now. |
jrosdahl
changed the title
-MD
jrosdahl
changed the title
|
Thanks! I always falsely assumed that the hardlink mode only influences files handed out of the cache (and I made sure we do not mess with those). That it also influences how files get into the cache was news to me. :-) |
How to reproduce
a.cppa.hpp:b.hpp// hi there!b.hppfile, remove the relevant include ina.hppand compile again. Ccache considers this a direct hit, handing out the cached object file and a bogusa.mfile, still referencing the deletedb.hppfile.Please note that the steps above are a minimal contrived example of what I think is going on, see below for a full log of a real instance of this issue.
Actual behavior
Here is a log of the second compilation of a source file as triggered by our build system, after switching branches in git to a state where a header was removed as in the example above:
Expected behavior
Per log output from above:
Ccache should detect a cache miss and compile the code again.
Environment
ccache version 3.7.9
The text was updated successfully, but these errors were encountered: