codeql action for cpp fails while creating the codeql db

[nnWhisperer]

Hello,
I searched with different keywords here and google and codeql/codeql repository but couldn't find a hinting place. The codeql analysis for cpp fails like below as taken from the github actions output:

    Finalizing cpp
      /home/anAdminForGithubVM/actions-runner/_work/_tool/CodeQL/2.14.3/x64/codeql/codeql database finalize --finalize-dataset --threads=8 --ram=29817 /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/cpp
      Running TRAP import for CodeQL database at /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/cpp...
      A fatal error occurred: Could not create array folder /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/cpp/db-cpp/default/idPool/indices3
      (eventual cause: FileSystemException "/home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/cpp/db-cpp/...")
      A fatal error occurred: Dataset import for /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/cpp/db-cpp failed with code 2.
      Error: Encountered a fatal error while running "/home/anAdminForGithubVM/actions-runner/_work/_tool/CodeQL/2.14.3/x64/codeql/codeql database finalize --finalize-dataset --threads=8 --ram=29817 /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/cpp". Exit code was 2 and error was: A fatal error occurred: Dataset import for /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/cpp/db-cpp failed with code 2.
      Context:
      A fatal error occurred: Could not create array folder /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/cpp/db-cpp/default/idPool/indices3
      (eventual cause: FileSystemException "/home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/cpp/db-cpp/...")

I'm trying to run codeql on the repository of a 3d open source software called blender. Previously, attempting to use ordinary github runners gave lack of disk space error on codeql action, so I set up an Azure cloud instance (Ubuntu-22.04) with 8 cpu and 32 Gb of Ram as a self-hosted runner. I set it up and started it in a tmux session with ./run.sh and saw that it was ready and working.
The error doesn't make sense since there were many other files created and have a huge disk space(300Gb).
After the error it stopped on the lines above. Their prepending lines were:

Run github/codeql-action/analyze@v[2](https://github.com/nnWhisperer/blender/actions/runs/6173419795/job/16755922895#step:7:2)
  with:
    category: /language:cpp
    output: ../results
    upload: always
    cleanup-level: brutal
    add-snippets: false
    skip-queries: false
    checkout_path: /home/anAdminForGithubVM/actions-runner/_work/blender/blender
    upload-database: true
    wait-for-processing: true
    token: ***
    matrix: {
    "language": "cpp"
  }
    expect-error: false
  env:
    CODEQL_ACTION_FEATURE_MULTI_LANGUAGE: false
    CODEQL_ACTION_FEATURE_SANDWICH: false
    CODEQL_ACTION_FEATURE_SARIF_COMBINE: true
    CODEQL_ACTION_FEATURE_WILL_UPLOAD: true
    CODEQL_ACTION_VERSION: 2.21.5
    JOB_RUN_UUID: 00a4c892-b847-40c5-8f09-07885f85[3](https://github.com/nnWhisperer/blender/actions/runs/6173419795/job/16755922895#step:7:3)720
    CODEQL_ACTION_ANALYSIS_KEY: .github/workflows/codeql-cpp-self-hosted.yml:analyze
    CODEQL_WORKFLOW_STARTED_AT: 2023-09-13T1[4](https://github.com/nnWhisperer/blender/actions/runs/6173419795/job/16755922895#step:7:4):19:39.432Z
    CODEQL_RAM: 29817
    CODEQL_THREADS: 8
    CODEQL_EXTRACTOR_KOTLIN_OVERRIDE_MAXIMUM_VERSION_LIMIT: 1.9.20
    CODEQL_EXTRACTOR_PYTHON_DISABLE_LIBRARY_EXTRACTION: true
    CODEQL_TRACER_LOG: /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/log/build-tracer.log
    CODEQL_TRACER_DIAGNOSTICS_DIR: /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/diagnostic/tracer
    CODEQL_TRACER_LANGUAGES: cpp
    SEMMLE_PRELOAD_libtrace: /home/anAdminForGithubVM/actions-runner/_work/_tool/CodeQL/2.14.3/x[6](https://github.com/nnWhisperer/blender/actions/runs/6173419795/job/16755922895#step:7:6)4/codeql/tools/linux64/${LIB}_${PLATFORM}_trace.so
    SEMMLE_PRELOAD_libtrace32: /home/anAdminForGithubVM/actions-runner/_work/_tool/CodeQL/2.14.3/x64/codeql/tools/linux64/lib32trace.so
    SEMMLE_PRELOAD_libtrace64: /home/anAdminForGithubVM/actions-runner/_work/_tool/CodeQL/2.14.3/x64/codeql/tools/linux64/lib64trace.so
    CODEQL_PLATFORM: linux64
    CODEQL_DIST: /home/anAdminForGithubVM/actions-runner/_work/_tool/CodeQL/2.14.3/x64/codeql
    CODEQL_SCRATCH_DIR: /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/working
    CODEQL_PLATFORM_DLL_EXTENSION: .so
    CODEQL_JAVA_HOME: /home/anAdminForGithubVM/actions-runner/_work/_tool/CodeQL/2.14.3/x64/codeql/tools/linux64/java
    CODEQL_EXTRACTOR_CPP_ROOT: /home/anAdminForGithubVM/actions-runner/_work/_tool/CodeQL/2.14.3/x64/codeql/cpp
    CODEQL_EXTRACTOR_CPP_WIP_DATABASE: /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/cpp
    CODEQL_EXTRACTOR_CPP_DIAGNOSTIC_DIR: /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/cpp/diagnostic/extractors/cpp
    CODEQL_EXTRACTOR_CPP_LOG_DIR: /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/cpp/log
    CODEQL_EXTRACTOR_CPP_SCRATCH_DIR: /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/cpp/working
    CODEQL_EXTRACTOR_CPP_TRAP_DIR: /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/cpp/trap/cpp
    CODEQL_EXTRACTOR_CPP_SOURCE_ARCHIVE_DIR: /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/cpp/src
    CODEQL_EXTRACTOR_CPP_THREADS: 8
    CODEQL_EXTRACTOR_CPP_RAM: 2981[7](https://github.com/nnWhisperer/blender/actions/runs/6173419795/job/16755922895#step:7:7)
    LD_PRELOAD: /home/anAdminForGithubVM/actions-runner/_work/_tool/CodeQL/2.[14](https://github.com/nnWhisperer/blender/actions/runs/6173419795/job/16755922895#step:7:14).3/x64/codeql/tools/linux64/${LIB}_${PLATFORM}_trace.so
    CODEQL_RUNNER: /home/anAdminForGithubVM/actions-runner/_work/_tool/CodeQL/2.14.3/x64/codeql/tools/linux64/runner
/home/anAdminForGithubVM/actions-runner/_work/_tool/CodeQL/2.14.3/x64/codeql/codeql version --format=terse
2.14.3

When I look at the same event on logs left on the self hosted server, it says the following:

    [2023-09-13 15:25:59Z INFO ExecutionContext] Publish step telemetry for current step {
      "action": "github/codeql-action/analyze",
      "ref": "v2",
      "type": "node16",
      "stage": "Main",
      "stepId": "657e1007-ff93-5216-d6a4-b6e19df1982b",
      "stepContextName": "__github_codeql-action_2",
      "hasPreStep": false,
      "hasPostStep": true,
      "result": "failed",
      "errorMessages": [
        "Encountered a fatal error while running \"/home/anAdminForGithubVM/actions-runner/_work/_tool/CodeQL/2.14.3/x64/codeql/codeql database finalize --finalize-dataset --threads=8 --ram=29817 /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/c"
      ],
      "executionTimeInSeconds": 670,
      "startTime": "2023-09-13T15:14:50.0719318Z",
      "finishTime": "2023-09-13T15:25:59.4846872Z"
    }.

1. I don't know whether it is related to what's causing my task to fail but, notice that it says "Encountered a fatal error while running "/home/anAdminForGithubVM/actions-runner/_work/_tool/CodeQL/2.14.3/x64/codeql/codeql database finalize --finalize-dataset --threads=8 --ram=29817 /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/c" Which is different from what the github actions UI shows. I guess that's a bug caused by referring to the same thing with different names "C" or "Cpp"
2. "/home/anAdminForGithubVM/actions-runner/_work/_temp/" is empty. To me it seems that the second bug is that the codeql_databases/* folders aren't created so it cannot write to subsequent folder. Of course these are only suspicions. So, please help. Thank you

[nnWhisperer]

https://github.com/nnWhisperer/blender/actions/runs/6173419795/job/16755922895
This is the job I'm referring to.
When I check that, codeql-action init step gave the following:

    /home/anAdminForGithubVM/actions-runner/_work/_tool/CodeQL/2.14.3/x64/codeql/codeql database init --db-cluster /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases --source-root=/home/anAdminForGithubVM/actions-runner/_work/blender/blender --language=cpp --begin-tracing --trace-process-name=Runner.Worker.exe --codescanning-config=/home/anAdminForGithubVM/actions-runner/_work/_temp/user-config.yaml --calculate-language-specific-baseline
    Calculating baseline information in /home/anAdminForGithubVM/actions-runner/_work/blender/blender
    Running command in /home/anAdminForGithubVM/actions-runner/_work/blender/blender: [/home/anAdminForGithubVM/actions-runner/_work/_tool/CodeQL/2.14.3/x64/codeql/javascript/tools/configure-baseline.sh]
    Calculated baseline information for languages: cpp, csharp, javascript, python, yaml (1.4s).
    Resolving extractor cpp.
    Successfully loaded extractor C/C++ (cpp) from /home/anAdminForGithubVM/actions-runner/_work/_tool/CodeQL/2.14.3/x64/codeql/cpp.
    Created skeleton CodeQL database at /home/anAdminForGithubVM/actions-runner/_work/_temp/codeql_databases/cpp. This in-progress database is ready to be populated by an extractor.

So it says that a skeleton CodeQL db was created in that directory but in fact it isn't.

[smowton]

I note the message You are running out of disk space. The runner will stop working when the machine runs out of disk space. Free space left: 0 MB

Can you allocate the VM a larger disk?

[nnWhisperer]

Hello @smowton, you might be looking at a previous run. In the last run, it gave error I mentioned. As I explained, I had already allocated an Azure gloud instance with 300Gb disk space. At the end only 35Gb were used.

[smowton]

Both https://github.com/nnWhisperer/blender/actions/runs/6173419795/attempts/1 and https://github.com/nnWhisperer/blender/actions/runs/6173419795/attempts/2 show a disk space related message. You mentioned that the database is gone at the end, presumably as a result of temp dir cleanout? Do you know that the disk didn't overflow during execution, only ending up at 35Gb after the temp dir was cleaned?