Ruby scanning job hangs forever and doesn't complete on Ubuntu-latest

[jedrekdomanski]

Hello,

We have set up a CodeQL code scannig job in our Ruby project and it takes over 6 hours to run and never completes. I have tried using both the default queries as well as security-extended and security-and-quality but they hang forever and never complete. We run two jobs (for Ruby and Javascript) using a language matrix. This is our codeql-analysis.yml file. Currently the timeout-minutes is set to 25 but it is only so to limit the run time and cut the cost of the job because we pay for it but it never completes. It was set to 6 hours but it didn't complete either.

name: "CodeQL"

on:
  push:
    branches: [ "master" ]
  pull_request:
    # The branches below must be a subset of the branches above
    branches: [ "master" ]

jobs:
  analyze:
    name: Analyze
    runs-on: ubuntu-latest
    timeout-minutes: 25
    permissions:
      actions: read
      contents: read
      security-events: write

    strategy:
      fail-fast: false
      matrix:
        language: [ 'ruby', 'javascript' ]
        # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
        # Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support

    steps:
    - name: Checkout repository
      uses: actions/checkout@v3

    # Initializes the CodeQL tools for scanning.
    - name: Initialize CodeQL
      uses: github/codeql-action/init@v2
      with:
        languages: ${{ matrix.language }}
        # If you wish to specify custom queries, you can do so here or in a config file.
        # By default, queries listed here will override any specified in a config file.
        # Prefix the list here with "+" to use these queries and those in the config file.

        # Details on CodeQL's query packs refer to : https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
        # queries: security-extended,security-and-quality



    # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
    # If this step fails, then you should remove it and run the build manually (see below)
    - name: Autobuild
      uses: github/codeql-action/autobuild@v2

    # ℹ️ Command-line programs to run using the OS shell.
    # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun

    #   If the Autobuild fails above, remove it and uncomment the following three lines.
    #   modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance.

    # - run: |
    #   echo "Run, Build Application using script"
    #   ./location_of_script_within_repo/buildscript.sh

    - name: Perform CodeQL Analysis
      uses: github/codeql-action/analyze@v2
      with:
        category: "/language:${{matrix.language}}"

Here is some logs, as you can see it just seats there and does not progress at all.

Starting evaluation of codeql/ruby-queries/queries/security/cwe-020/IncompleteUrlSubstringSanitization.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-020/MissingRegExpAnchor.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-020/OverlyLargeRange.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-022/PathInjection.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-078/CommandInjection.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-078/KernelOpen.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-078/NonConstantKernelOpen.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-078/UnsafeShellCommandConstruction.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-079/ReflectedXSS.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-079/StoredXSS.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-079/UnsafeHtmlConstruction.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-089/SqlInjection.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-094/CodeInjection.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-094/UnsafeCodeConstruction.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-116/BadTagFilter.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-116/IncompleteMultiCharacterSanitization.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-116/IncompleteSanitization.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-117/LogInjection.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-1333/PolynomialReDoS.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-1333/ReDoS.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-1333/RegExpInjection.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-134/TaintedFormatString.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-209/StackTraceExposure.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-295/RequestWithoutValidation.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-300/InsecureDependencyResolution.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-312/CleartextLogging.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-312/CleartextStorage.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-327/BrokenCryptoAlgorithm.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-352/CSRFProtectionDisabled.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-502/UnsafeDeserialization.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-506/HardcodedDataInterpretedAsCode.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-598/SensitiveGetQuery.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-601/UrlRedirect.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-611/Xxe.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-732/WeakCookieConfiguration.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-798/HardcodedCredentials.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-829/InsecureDownload.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-912/HttpToFileAccess.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-918/ServerSideRequestForgery.ql.
Starting evaluation of codeql/ruby-queries/queries/summary/LinesOfCode.ql.
[3/46 eval 2.4s] Evaluation done; writing results to codeql/ruby-queries/queries/summary/LinesOfCode.bqrs.
Starting evaluation of codeql/ruby-queries/queries/summary/LinesOfUserCode.ql.
Starting evaluation of codeql/ruby-queries/queries/summary/NumberOfFilesExtractedWithErrors.ql.
[4/46 eval 8ms] Evaluation done; writing results to codeql/ruby-queries/queries/summary/LinesOfUserCode.bqrs.
Starting evaluation of codeql/ruby-queries/queries/summary/NumberOfSuccessfullyExtractedFiles.ql.
[5/46 eval 3ms] Evaluation done; writing results to codeql/ruby-queries/queries/summary/NumberOfFilesExtractedWithErrors.bqrs.
[6/46 eval 5ms] Evaluation done; writing results to codeql/ruby-queries/queries/summary/NumberOfSuccessfullyExtractedFiles.bqrs.
[7/46 eval 1m16s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-020/IncompleteUrlSubstringSanitization.bqrs.
[8/46 eval 1m30s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-300/InsecureDependencyResolution.bqrs.
[9/46 eval 1m33s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-352/CSRFProtectionDisabled.bqrs.
[10/46 eval 1m36s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-732/WeakCookieConfiguration.bq

[aibaars]

@jedrekdomanski Thanks for reporting. It looks like your repository somehow runs into a performance issue with one of the queries. Did things work for you in the past or did you just setup CodeQL analysis for your repository.

If things used to work, could you try running a previous version of CodeQL as a workaround. This can be done by setting the tools: property of the github/codeql-action/init to the download URL of codeql-bundle-linux64.tar.gz of an earlier release https://github.com/github/codeql-action/releases .

Could you try :

   - name: Perform CodeQL Analysis
      uses: github/codeql-action/analyze@v2
      with:
        category: "/language:${{matrix.language}}" 
      env:
        CODEQL_ACTION_EXTRA_OPTIONS: '{ "database": { "analyze": ["--timeout", "600"] } }'

This should limit runs of codeql database analyze to about 10 minutes for a query. This way the log should inform us which query it was working on when the timeout was reached. Could you also re-run the workflow with the debug logging: https://github.blog/changelog/2022-05-24-github-actions-re-run-jobs-with-debug-logging/ ? That should collect more detailed logs and also save intermediate artifacts such as the CodeQL database.

If this is an open source repository, could you share the URL and any debug artifact so we can investigate.

If this is a closed source repository, please contact GitHub support to continue this conversation via an internal support ticket.

[jedrekdomanski]

Thank you for your quick reply. It was never successful before, we've only just started running the scannig jobs in the project. I'll try your suggestions. In the meantime, here's the full log of the job which I ran in debug mode.
https://pipelines.actions.githubusercontent.com/serviceHosts/0887281a-bad0-42c6-a967-808339f1591c/_apis/pipelines/1/runs/4450/signedlogcontent/3?urlExpires=2023-03-01T12%3A57%3A59.3172412Z&urlSigningMethod=HMACV1&urlSignature=hq6k5n8uXCj3hMj8Dt0vXVrmPx%2FC3BJf%2FE3R0AU0bqw%3D

[aibaars]

Thanks for the quick reply, unfortunately the URL you posted had expired before I could download it.

[jedrekdomanski]

I've attached the logs of a failed job below.
output.txt

[jedrekdomanski]

Here is another output file of a job that's just failed.
output.txt

[aibaars]

Looking at the output of the "resolve files" command, it seems like your repository is quite large. Most likely CodeQL is running low on memory due to the size of the repository which causes it to slow down. You could try running the analysis on a larger runner or a self-hosted one: Using larger runners.
See also: Recommended hardware resources for running CodeQL

Another thing to try is to reduce the number of scanned files. The spec folder is probably test cases, and most likely do not need to be scanned for security vulnerabilities. You could also exclude the db migrations. See also: Specifying directories to scan

If reducing the files and increasing the RAM does not work then it would be helpful to do the following:

• checkout the source code locally
• download the CodeQL CLI: https://github.com/github/codeql-cli-binaries/releases
• run: codeql database create -lruby -s checkout_folder /tmp/database-folder
• run: codeql database analyze --tuple-counting --evaluator-log=/tmp/evaluator.log /tmp/database-folder
• attach the evaluator.log and database-folder/logs folder (zipped)

[jedrekdomanski]

@aibaars I've tried reducing the number of scanned files but this doesn't work. The documentation says to add this:

paths:
  - src
paths-ignore:
  - src/node_modules
  - '**/*.test.js'

And so says the example config file here https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/customizing-code-scanning#example-configuration-files

name: "My CodeQL config"

disable-default-queries: true

queries:
  - name: Use an in-repository QL pack (run queries in the my-queries directory)
    uses: ./my-queries
   ...
paths:
  - src 
paths-ignore: 
  - src/node_modules
  - '**/*.test.js'

So I added this to the root namespace in my config file:

name: "CodeQL"

on:
  push:
    branches: [ "master" ]
  pull_request:
    # The branches below must be a subset of the branches above
    branches: [ "master" ]

jobs:
  ...

paths:
  - app
  - lib
paths-ignore: 
  - spec
  - db

but I get an error

The workflow is not valid. .github/workflows/codeql-analysis.yml (Line: 74, Col: 1): Unexpected value 'paths' .github/workflows/codeql-analysis.yml (Line: 77, Col: 1): Unexpected value 'paths-ignore'

[aibaars]

I think you need to put the configuration in a separate file and refer to it using the config-file: property.

- name: Perform CodeQL Analysis
   uses: github/codeql-action/analyze@v2
   with:
     category: "/language:${{matrix.language}}" 
     config-file: ./.github/codeql/codeql-config.yml
   env:
     CODEQL_ACTION_EXTRA_OPTIONS: '{ "database": { "analyze": ["--timeout", "600"] } }'

See Working with custom configuration files

[jedrekdomanski]

Unexpected input(s) 'config-file', valid inputs are ['check_name', 'output', 'upload', 'cleanup-level', 'ram', 'add-snippets', 'skip-queries', 'threads', 'checkout_path', 'ref', 'sha', 'category', 'upload-database', 'wait-for-processing', 'token', 'matrix', 'expect-error']

So I am surprised... that I do what the docs say and it doesn't work.
But clearly it is a performance problem so the question is how do I reduce the number of scanned files? :)

[aibaars]

I'm sorry, the yaml snippet I included is wrong. The config-file property should be set on the github/codeql-action/init step.

- uses: github/codeql-action/init@v2
  with:
    config-file: ./.github/codeql/codeql-config.yml

[jedrekdomanski]

Is this a correct configuration?

# Initializes the CodeQL tools for scanning.
    - name: Initialize CodeQL
      uses: github/codeql-action/init@v2
      with:
        languages: ${{ matrix.language }}
        config-file: ./.github/codeql/codeql-config.yml

config file

# ./.github/codeql/codeql-config.yml

paths:
  - app
  - lib
paths-ignore:
  - spec
  - db

[jedrekdomanski]

Anyway, it doesn't seem to improve the runtime at all and env property

env:
     CODEQL_ACTION_EXTRA_OPTIONS: '{ "database": { "analyze": ["--timeout", "600"] } }'

doesn't look like it has any effect.
Also, I tried setting the previous version of CodeQl as you suggested in your first reply using tools property on the github/codeql-action/init step pointing to the URL of the previous version but it doesn't work.

# Initializes the CodeQL tools for scanning.
    - name: Initialize CodeQL
      uses: github/codeql-action/init@v2
      tools: /github/codeql-action/releases/download/codeql-bundle-20221024/codeql-bundle-linux64.tar.gz
      with:
        languages: ${{ matrix.language }}
        config-file: ./.github/codeql/codeql-config.yml

The workflow is not valid. .github/workflows/codeql-analysis.yml (Line: 45, Col: 7): Unexpected value 'tools'

[aibaars]

env:
     CODEQL_ACTION_EXTRA_OPTIONS: '{ "database": { "analyze": ["--timeout", "600"] } }'

doesn't look like it has any effect.

My bad, I thought the codeql-action ran the codeql database analyze command but it looks like it is running the lower level database run-queries command instead: https://github.com/github/codeql-action/blob/a589d4087ea22a0a48fc153d1b461886e262e0f2/src/codeql.ts#L820

Could you try:

env:
     CODEQL_ACTION_EXTRA_OPTIONS: '{ "database": { "run-queries": ["--timeout", "600"] } }'

[aibaars]

Also, I tried setting the previous version of CodeQl as you suggested in your first reply using tools property on the github/codeql-action/init step pointing to the URL of the previous version but it doesn't work.

The tools property should be under the with: section (and include https://).

# Initializes the CodeQL tools for scanning.
    - name: Initialize CodeQL
      uses: github/codeql-action/init@v2
      with:
        tools: https:/github/codeql-action/releases/download/codeql-bundle-20221024/codeql-bundle-linux64.tar.gz
        languages: ${{ matrix.language }}
        config-file: ./.github/codeql/codeql-config.yml