Perform CodeQL Analysis on C++ code fails with "Oops! A fatal internal error occurred."

[geoffthemedio]

When testing CodeQL on my project's C++ code, the build succeeded but the following analysis step failed. The error in the log has a lot of Java-related details that are perhaps too long to paste here, but it starts with:

Oops! A fatal internal error occurred.
  java.lang.RuntimeException: com.google.common.util.concurrent.ExecutionError: java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
  	at com.semmle.util.exception.Exceptions.asUnchecked(Exceptions.java:103)
  	at com.semmle.inmemory.trap.TrapImporter.run(TrapImporter.java:106)
  	at com.semmle.cli2.ql.dataset.ImportCommand.executeSubcommand(ImportCommand.java:86)
  	at com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:110)
  	at 
[...]

The test can be seen here: https://github.com/freeorion/freeorion/pull/3325/checks?check_run_id=1965808793

Googling the error lead to an issue that suggested making a report such as this.

[alexet]

Thanks for the report. This is a badly reported out of disk space error.

Sometimes clearing the build artefacts after the build but before analysis will get you enough space if your build artefacts take up a lot of space. Building less can also help although it looks like you aren't building any third party things. Otherwise you may have to use a machine with more disk space.

[geoffthemedio]

Sometimes clearing the build artefacts after the build but before analysis will get you enough space if your build artefacts take up a lot of space.

Could you point me to instructions for how to do that?

Building less can also help although it looks like you aren't building any third party things.

It might be the template-heavy Boost-dependent code.

Otherwise you may have to use a machine with more disk space.

Is that something I can control with GitHub Actions CodeQL? Or do you mean it must be run offline instead?

[alexet]

Looking into this a bit further I think the issue is just the template heavy code. We record data about all template instatiations which takes up a lot of space. It looks like you need about 20-25GB of space to run codeql on your codebase.

Sadly I don't think this is going to work on the standard runners as they only have 17GB. You could run the analysis offline.

[geoffthemedio]

Does it need to link the final libraries and binaries to be able to run the analysis?

If not, I could exclude some subprojects or source files, such as the very template-heavy resource-intensive parsers, and still hopefully get useful results about the rest of the codebase.

[alexet]

We can get better results if we see the linker execution as we can disambiguate calls where linking means only one target is available but if everything is linked together anyway it is usually less important as no disambiguation is needed.

[geoffthemedio]

I modified the .yml to set some defines in the CMake call to exclude building several projects. There's no pre-made define to exclude the template-heavy parsers, but nonetheless this seems to have allowed the analysis to run partially. However, it still eventually runs out of disk space, see: https://github.com/freeorion/freeorion/runs/1974640186?check_suite_focus=true#step:7:129

I'd like to try the suggestion to "clearing the build artefacts after the build", but I'm not sure how to do that. Googling leads to various scripts for actions for cleaning up stale artifacts, which seem to be intended to be run periodically as a separate action, but I presumably need to do this during the CodeQL action itself. Could you point me to instructions or an example of what to add to the .yml to make that happen?

Also, is there a way to view the results of the analysis that ran partially, or perhaps were necessary subsequent steps not run after the out of disk failures?

[alexet]

The "clearing the build artefacts after the build" suggestion was something like doing rm -rf build at the end of the of the build as we don't care about the build results. However I hadn't looked in detail and hadn't noticed that you only had about 200MB to spare so it might not gain you much.

Unfortunately there isn't really a way of seeing the partial results as we need to post process the query results.

[geoffthemedio]

rm -rf build didn't help, but I was able to get it to run through and generate a single useful warning from the analysis by excluding some source files from the build.