Perform Codeql analysis error

[Subharanjan2055]

While performing codeql analysis am getting below error
A fatal error occurred: Dataset import for /home/devops/actions-runner-uat/_work/_temp/codeql_databases/javascript/db-javascript failed with code 137.
Error: The process '/home/devops/actions-runner-uat/_work/_tool/CodeQL/0.0.0-20220615/x64/codeql/codeql' failed with exit code 2
it's a python based application with some javascript lib files , while performing code-scanning python is working fine but when it comes to javascript it fails .
To perform this scanning am using linux based self-hosted runner .

I dont have idea why does this happens. please let me know any solutions.

[hmakholm]

Exit code 137 means the process was terminated by receiving a SIGKILL signal. Unless you're doing kill -9 yourself from a shell, the usual culprit is the "Linux OOM killer": the kernel will randomly kill a memory-heavy process when the total RAM demand from all processes exceeds the physical RAM plus swap space it has available.

Unfortunately there's no direct way to control how much RAM the "dataset import" phase of database construction will take. Analysis of larger codebases will just need proportionally more RAM. If you're using a threads parameter to make the import happen in multiple threads, doing it single-threaded can improve it somewhat.

Also check if the machine is running other RAM-heavy things in parallel -- for example if the Python analysis is still running, or if your self-hosted runner runs different tasks together.

[Subharanjan2055]

@hmakholm am using github action for codeql. Is there any way to control memory usage via action workflow script

[hmakholm]

Unfortunately there's no current way to configure this phase to use less memory.

If you were running the CodeQL CLI directly rather than through codeql-action, you could give an -J-Xmx<megabytes> option to the codeql database create command to set the heap space limit for the Java virtual machine that's excuting the CodeQL engine.

Without this option, the JVM will default to using up to 1/4 of whatever the OS says the physical memory size is, so if it's the only thing that is running, you shouldn't be seeing into OOM killer trouble. If you're running the self-hosted runner in a virtualized environment, the memory-size reporting might be sufficiently off to fool Java, but I haven't heard of that being a common problem.

[hmakholm]

Unfortunately there's no current way to configure this phase to use less memory.

Scratch that -- The relevant engineering team tells me that we do have an undocumented way to do this: In the workflow block that runs the github/codeql-action/analyze action, add

    env:
      CODEQL_ACTION_EXTRA_OPTIONS: '{"database": {"finalize": ["-J-Xmx1234"]}}'

replacing 1234 with a number of megabytes. (This controls just the heap size of the CodeQL process; a few hundred megabytes might additionally be used for other purposes).

Some trial and error may be necessary to find a value that has a chance of working -- it you set it too low for the codebase you're analyzing, the same phase may instead terminate with an explicit, internally generated "out of memory" condition.

[vladimirr9]

Correct me if I'm wrong but for -Xmx1234 the value set by default is in bytes, not megabytes.
If we wanted to set megabytes it would be -Xmx1234m.
At least that's what I had to do to avoid the "out of memory" termination you mentioned.

[aeisenberg]

If we wanted to set megabytes it would be -Xmx1234m.

That's correct.

[cooljeanius]

Correct me if I'm wrong but for -Xmx1234 the value set by default is in bytes, not megabytes. If we wanted to set megabytes it would be -Xmx1234m. At least that's what I had to do to avoid the "out of memory" termination you mentioned.

Could I get some clarification as to who "we" refers to here? Are users supposed to set this flag, or is it supposed to be done in the action?

[aeisenberg]

The CodeQL Action has its own way of determining how much memory to use, based on the total amount of memory in the system. in 99.5% of the cases, this is good enough and you don't need to set the memory yourself explicitly.

If you are running out of memory, you can set the CODEQL_ACTION_EXTRA_OPTIONS environment variable. This should be a very rare occurrence.

So, in this case "we" refers to the user.

[cooljeanius]

The CodeQL Action has its own way of determining how much memory to use, based on the total amount of memory in the system. in 99.5% of the cases, this is good enough and you don't need to set the memory yourself explicitly.

If you are running out of memory, you can set the CODEQL_ACTION_EXTRA_OPTIONS environment variable. This should be a very rare occurrence.

I tried doing this in cooljeanius/apple-gdb-1824@b559d98, but it doesn't seem to have worked...

[cooljeanius]

The CodeQL Action has its own way of determining how much memory to use, based on the total amount of memory in the system. in 99.5% of the cases, this is good enough and you don't need to set the memory yourself explicitly.
If you are running out of memory, you can set the CODEQL_ACTION_EXTRA_OPTIONS environment variable. This should be a very rare occurrence.

I tried doing this in cooljeanius/apple-gdb-1824@b559d98, but it doesn't seem to have worked...

Update: looks like doubling the CODEQL_RAM value doesn't work either:
https://github.com/cooljeanius/apple-gdb-1824/actions/runs/7902634952/job/21571629581
I'll try using the number listed as the amount available next...

[aeisenberg]

The image you are using only has 16GB of ram, so adding the extra option bumped the requested amount of ram over the physical ram that's available.

You could try using an xl sized runner.