-MQ gcc option breaks "compiling same file" heuristic

[Hi-Angel]

-MQ path is one of options that mesonbuild system always adds. It makes ccache to not notice that the same file is being compiled, so basically that affects all projects that are using mesonbuild.

Steps to reproduce (in terms of terminal commands)

$ cat slow.cpp
template <int i>
class  A {
    A<i-1>   x;
    A<i-2>   y;
};

template <> class A<0> {
    char a;
};

template <> class A<1> {
    char a;
};

void slow() {
    A<35> b;
}
$ ccache -C   
Cleared cache
$ time ccache c++ -g -MD -MQ slow1.cpp.o -o slow.cpp.o -c slow.cpp
ccache c++ -g -MD -MQ slow1.cpp.o -o slow.cpp.o -c slow.cpp
 3.18s user 0.04s system 99% cpu 3.220 total
$ time ccache c++ -g -MD -MQ slow2.cpp.o -o slow.cpp.o -c slow.cpp
ccache c++ -g -MD -MQ slow2.cpp.o -o slow.cpp.o -c slow.cpp
 3.18s user 0.02s system 99% cpu 3.200 total

Expected

The second compilation should've taken much less time

Actual

Both compilations take the same time.

Version

ccache 3.6

[nirbheek]

Is there any difference if you use -MT instead of -MQ?

[Hi-Angel]

Is there any difference if you use -MT instead of -MQ?

No difference, same compilation time.

[nirbheek]

I can't actually reproduce this:

$ time ccache c++ -g -MD -MQ slow1.cpp.o -o slow.cpp.o -c slow.cpp
real    0m3.019s
user    0m2.969s
sys     0m0.029s
$ time ccache c++ -g -MD -MQ slow1.cpp.o -o slow.cpp.o -c slow.cpp
real    0m0.009s
user    0m0.001s
sys     0m0.009s
$ ccache --version
ccache version 3.4.2

[Hi-Angel]

You didn't change the path in the second compilation that goes after -MQ

[nirbheek]

Oh, then I don't understand the problem. ccache only looks at the command-line being passed, it does not parse it for anything other than the depfile, afaik. What do you expect it to do?

[Hi-Angel]

Well, I'm confused. The manual says

ccache is a compiler cache. It speeds up recompilation by caching the result of previous compilations and detecting when the same compilation is being done again. Supported languages are C, C++, Objective-C and Objective-C++.

From my point of view compiling the same file is "the same compilation being done". I may understand incorrectly what ccache does, then maybe it worth to be more explicit in the man page…

[Hi-Angel]

Btw, before reporting the bug I even looked at the "caveats" link in the manual, so it's not like I didn't look at docs.

[nirbheek]

When you change the -MQ argument, the depfile has to be rewritten AIUI, so you have to call the compiler again. I may be wrong about this, and perhaps ccache can do some optimization, but it seems like normal behaviour. How does this affect your usage of Meson?

[Hi-Angel]

How does this affect your usage of Meson?

Not my question, but summarizes that well

[nirbheek]

The answer selected for that question is correct. Another way is to have two targets share the same objects is with extract_all_objects(). I'll stop commenting here about Meson things now because it's probably noise for the ccache devs ;)

[jrosdahl]

From my point of view compiling the same file is "the same compilation being done". I may understand incorrectly what ccache does

@nirbheek is correct. "The same compilation is being done" in ccache's case means "produces the same output", where output is "stdout/stderr and all files created by the compilation". Since the two commands in your example don't produce the same .d file content, ccache treats the compilations as separate so as not to produce false positive cache hits.

It should however be possible to teach ccache to understand what -MT and -MQ do and adjust the .d file appropriately when copying the result from the cache (and not include -MT/-MQ options and their arguments in the hash, of course).

then maybe it worth to be more explicit in the man page…

That could be arranged. Do you have a suggestion of a formulation that would have made sense to you?

[Hi-Angel]

That could be arranged. Do you have a suggestion of a formulation that would have made sense to you?

Well, if that can't be worked around, I think something like the following could be added to Caveats section (I may use incorrect terms there as it's the first time I learned about dependencies files):

• "Same file re-compilation" with different from previous compilations target name inside dependency file is being counted as a completely new compilation. This happens e.g. if you override the name with -MT or -MQ gcc and clang options. Meson build system in particular is known to implicitly add these to their distinct build targets.

[jrosdahl]

Well, if that can't be worked around

I think that such a feature (I wouldn't call it a workaround) would be possible, see #359.

I think something like the following could be added to Caveats section (I may use incorrect terms there as it's the first time I learned about dependencies files):

The Caveats section in the manual describes cases where the statement "ccache [always produces] exactly the same compiler output that you would get without the cache" does not hold. I'd prefer not to add other kinds of information there.

The Limitations section (both in the manual and on the front web page) already says "Some compiler flags are not supported. If such a flag is detected, ccache will silently fall back to running the real compiler". This is sort of what happens here, except that the flag itself is supported but its semantics forces ccache to add extra information to the hash to distinguish the compilations. There are probably hundreds of other cases (both known and unknown) with flags that change the ccache caching behavior. For instance, these two commands without -MQ but with different .o files will also be seen as different:

ccache c++ -g -MD -o slow1.cpp.o -c slow.cpp
ccache c++ -g -MD -o slow2.cpp.o -c slow.cpp

The reason is the same: the .d file content is not the same in the two cases.

What I'm looking for is to improve the documentation in a more generic way so that you and others won't be surprised that there are scenarios where ccache will include potentially surprising bits of information in the hash. I'm afraid that adding too specific documentation such as referring to Meson will have too high risk of becoming outdated.

[Hi-Angel]

I see, thanks! So, I didn't see Limitations section. No wonder here, the documentation is pretty big, so I read just the general description and Caveats, and thought that I got everything I need, since I didn't need run modes, options, configuration, etc.

But Limitations section is at the very beginning, how could I miss that! I retraced my thought process, I think what's happening is this: I read Description section till the end, there I see next section Features. I read its first sentence, conclude it's a common section of most docs of how cool the software being described, and that I don't need this right now, and stop reading there because usually after that goes description of options. But Description has a link to Caveats, so I navigate there, don't see anything remotely similar to my situation, and stop at that.

I don't know how common this thought process, and hence whether modifying docs would make sense in general. But if a question is how I personally wouldn't miss Limitations section: I wouldn't if either α) Limitations were before Features, or β) Description has a link to Limitations subsection.

In general, to me it seems rational if Limitations and Caveats sections would go one after another, as in "a list of possible issues in a single place".

[jrosdahl]

I've made some minor changes to the Limitations and Is it safe? sections now, but I decided to keep the overall structure of the main page since I don't know how improve clarity without sacrificing some other aspect.