compile_commands.json is empty when build gcc 11.2.0 from soure

[jiangjianshan]

Describe the bug
The compile_commands.json is empty when I use Bear which version 3.0.15 when build gcc which version 11.2.0. The script I used is here:

To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.

Environment:

• OS name: Linux
• OS version: Kali Linux 2021.2
• OS architecture: x86_64
• Bear version: 3.0.15
• Bear install method: from package

Additional context

• Can you give us a reference to the project that you are running against this tool? gcc 11.2.0
• What build tools this project is using? GNU make
• What architecture you are trying to compile for? [e.g. cross compiling]
• Could you attach build logs? If you can re-run the command which failed (e.g. bear make) with extra verbose log switches (e.g. bear -vvvv make) and send the output of it, that would help a lot.

Before you start
end...

• Have you read the README.md file or man bear? Yes
• Have you looked what other open issues Bear has? Yes
• Are you sure that the build works without Bear? Yes

[rizsotto]

Hey @jiangjianshan , can you send the output of the build of this command: bear --help?

I am suspicious about the --library is pointing to a directory which does not have the libexec.so... I presume the Kali linux is a debian derivative, so the $LIB will expand to lib/x86_64-linux-gnu. Please also send the Bear package file list to see where the package installs the libexec.so file.

[jiangjianshan]

Hello @rizsotto, here is the command ouput:

Bear can successfully generate the compile_commands.json correctly when build the following packages from source:

1. bash-5.1.8
2. diffutils-3.8
3. gawk-5.1.0
4. binutils-2.37
5. emacs-27.2
6. global-6.6.7
7. linux-5.10.41
8. webbench-1.5
9. coreutils-8.32
10. findutils-4.8.0
11. guile-3.0.7
12. lua-5.4.3
    Only gcc-11.2.0 was failed to generate compile_commands.json which I have try at least three or four times.

[rizsotto]

Thanks @jiangjianshan , we can exclude the packaging problem then. :)

I would like to ask for more logs... Instead of executing bear --verbose -- make -j $(nproc), I would like to ask you to do this: intercept --verbose -- make -j $(nproc) and citnames --verbose. (This will split the work to intercept the command executions, and create the JSON compilation database.) It would be good to have the log from these commands. (You can attach those to this issue.) And the intercept command will create a file events.json, which would be also interesting, but can be big. (Maybe a wc -l events.json can do.)

From these information, we can see which steps fail and concentrate to fix that. I appreciate your help for the debug, because I don't know much about Kali Linux. Thanks!

[jiangjianshan]

@rizsotto , I have just run the commmands ./build-gcc.sh 2>&1 | tee build.log and I wait for it finish this time.
compile_commands.json.tar.gz
events.json.tar.gz
build.log.tar.gz
This time I have found compile_commands.json can be generated sucessfully. The change is only I use 'intercept --verbose -- make -j$(proc) and 'citnames --verbose' instead of 'bear --verbose -- make -j$(nproc)'.
But I have found that the finish of compile time is abnormal when build gcc from source. It seems like the compile hasn't been finished.

[jiangjianshan]

@rizsotto , I chang back the command to 'bear --verbose --make -j$(nproc)' and run ./build-gcc.sh 2>&1 | tee build.log again. I have found compile_commands.json can be generated but the size of it is a little small. and much small than the one generated by two commands you recommends. And the compile time is two short to build a complete gcc from source. It seems like abnormal.

compile_commands.json.tar.gz
build.log.tar.gz

[rizsotto]

No, this is fine. Thank you @jiangjianshan !

The compile_commands.json is not empty, but it's also not good enough. The entries in that file are mostly from compiler checks. And from the build log, I see very few intercepted compiler calls. (That explains the output is not complete.)

If the build is using a statically linked compiler, the intercept process is blind for that. Bear's intercept has two modes (as it is explained in the man intercept page), and the one you were trying is the "preload" one. Alternatively you can try the "wrapper" mode. That will require to run the configure step with Bear too, but ignore the output from that run. (Usually the "preload" has better results, but "wrapper" mode might fit here better.)

$ bear --force-wrapper --verbose -- ./configure ...
$ bear --force-wrapper --verbose -- make -j $(nproc)

[jiangjianshan]

@rizsotto , I have just try above new two commands you suggested. Then the full content of bash script is here:

Here is build logs:
build.log.tar.gz
Now, compile_commands.json was generated but the content is empty. Here is the gcc version I'm using.

[jiangjianshan]

@rizsotto , maybe this issue is not related to kali linux 2021.2. I have try different command combination still can not solved this issue. It should can be reproduce it when build gcc 11.2.0 on your using Linux distribution with Bear command. The gcc source I'm using is from https://ftp.gnu.org/gnu/gcc/gcc-11.2.0/gcc-11.2.0.tar.xz.

[rizsotto]

Thanks @jiangjianshan for your investigation. It might be that it is a problem that Bear can't address. (There are already a few, like bazel build, or rare build file encoding, etc...) This might be related to GCC build system, which can not be covered by Bear intercept modes. 🤷

Will record it as a limitation of this tool in the documentation. And will try to reproduce the issue on my machines.

I can't offer you solution nor workaround on this, I'm sorry.

[jiangjianshan]

@rizsotto , Thanks for your help previously. I have also try to build gcc 11.2.0 from source without Bear. The complete log are here. Maybe it can be helpful for your analysis in the future.
build.log.tar.gz
Bear is a good tool, maybe it need to support a configuration file or options to support the intercept process for those build process is no standard, for example using libtool on some parts:

[rizsotto]

Spent some time to reproduce this problem, but could not.

Tested on Fedora/Arch/Ubuntu. Added test case against libtool which runs fine on these distributions.

[jiangjianshan]

@rizsotto , thanks for your work. I will update the bear build when the release is ready.