v2.15.3 - Python code analysis fails #2042
Comments
|
A web search for |
|
Sorry, I forgot to mention that this job runs within a container using a custom image that had Python 3.11. I forgot to include the container part in my workflow file snippet (updated it now)
|
|
Maybe running the Lines 66 to 69 in 511f073
|
|
I'm not sure, but even in a container, the workflow will try to use the python version in the toolcache. Could you try explicitly setting up python 3.11? Something like this, before you run the |
|
Now that error is gone but I am presented with a different one |
|
Hmmm...have you tried turning off |
|
Just tried that by setting |
@aeisenberg Also, may I ask why is this the case? The TypeScript CodeQL scanner seems happy to use the Node version installed in the container, but the Python scanner doesn't seem to want to use it. |
|
For typescript, no compilation or code execution is required during extraction. For python, we need to execute the python extractor, which is built in python. @RasmusWL do you have any suggestions on what to do? |
|
@akr-amd The error message you are seeing now is caused by CodeQL not scanning any Python source files in the repository folder. Could you try running without the Another reason could be a mismatch between what CodeQL considers the "source root" and the path of the repository in the container. In that case CodeQL did scan files but would not count them before they are found in an "external" folder. Could you also re-run the workflow with debug logging enabled? That should result in a CodeQL debug artifact containing much more detailed logs and also any source files that CodeQL has picked up. Finally, there is no need to run Python or Typescript analysis in a docker container, so you could also try removing the |
|
The suggestions from @aibaars seems solid, let's see if those suggestions solves the problem 👍 |
|
At the outset, thanks for being super responsive and helpful! 🙂
Based on feedback, here's how I changed the workflow
The new workflow file is below # For most projects, this workflow file will not need changing; you simply need
# to commit it to your repository.
#
# You may wish to alter this file to override the set of languages analyzed,
# or to provide custom queries or build logic.
#
# ******** NOTE ********
# We have attempted to detect the languages in your repository. Please check
# the `language` matrix defined below to confirm you have the correct set of
# supported CodeQL languages.
#
name: "CodeQL"
on:
pull_request:
branches: [ "main" ]
push:
branches: [ "main" ]
jobs:
analyze:
name: Analyze
runs-on: [ self-hosted, Linux ]
permissions:
actions: read
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
language: [ 'javascript', 'python' ]
# CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
# Use only 'java' to analyze code written in Java, Kotlin or both
# Use only 'javascript' to analyze code written in JavaScript, TypeScript or both
# Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support
steps:
- name: Checkout repository
uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.11'
- uses: actions/setup-node@v3
with:
node-version: '16.15.1'
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v2
with:
languages: ${{ matrix.language }}
# config-file: ./.github/codeql/codeql-config.yml
setup-python-dependencies: false
# If you wish to specify custom queries, you can do so here or in a config file.
# By default, queries listed here will override any specified in a config file.
# Prefix the list here with "+" to use these queries and those in the config file.
# Details on CodeQL's query packs refer to : https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
# queries: security-extended,security-and-quality
# Autobuild attempts to build any compiled languages (C/C++, C#, Go, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
uses: github/codeql-action/autobuild@v2
# ℹ️ Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
# If the Autobuild fails above, remove it and uncomment the following three lines.
# modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance.
# - run: |
# echo "Run, Build Application using script"
# ./location_of_script_within_repo/buildscript.sh
- run: |
echo "$(which python)"
echo "$(python -V)"
echo $LGTM_INDEX_FILTERS
find . -name '*.py'
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v2
with:
category: "/language:${{matrix.language}}"Without providing the config file, it is able to find all the files. But that's the next problem. The and the
Not sure if I am doing something wrong, but the value of
Here's the output of
May I ask why the analysis shouldn't be run in a container? I was coming at it from the angle of 'pristine env of a container won't cause any env unintended cross-contamination issues' |
That is actually not a problem, but expected behaviour. Those JavaScript files from CodeQL contain stubs for JS functions that are available by default in (browser) environments. The Python analysis also scans the standard libraries to figure out dataflow through standard functions. CodeQL can stop doing that, once the team has implemented QL models for the standard libraries, but for now it is still needed.
Ah yes sorry, I should have made clear that I was interested in the value lf
I did not mean to say that you shouldn't run in a container. Having a "pristine" env can be quite beneficial. It's just that JS and Python analysis don't change the environment and should work fine in a not so "pristine" environment. The reason why I asked to run outside a container was to reduce complexity. When running in a container there is always the risk of mixing up file paths from the host or the container. Things should have worked fine with a configuration file and a container, but they didn't ;-) To debug it helps to disable those features to see which one causes the problem, or whether it is the combination of both. |
Oh ok got it. So, seems it's safe to ignore these.
Hmm, I used the config file (shown below) and re-ran the workflow, but paths:
- 'engine/**/*.py'
- 'cli/**/*.py'
- 'ui/**/*.ts'
Makes sense - Eliminate possible sources of problem |
|
I could reproduce, and narrowed down the problem to cases where (if analyzing |
Sorry, I thought I replied back. The suggested config file does work @RasmusWL. Thanks! |
|
👋 Team, I'm also facing similar issue while running codeql 2.15.3 version . Note I'm only testing a simple python codeql query . For other languages java/javascript same command is working fine. Command Output
|
|
Hi @BullHacks3, as already mentioned in this issue:
|
|
Hey @RasmusWL I'm already using python 3.11 version .Thanks |
|
@BullHacks3 please open a new issue then. |
|
Thanks @RasmusWL , created new issue for same : github/codeql#15337. Thanks |
Hi there, I am trying to setup CodeQL analysis on a repo in our github enterprise server. This is a monorepo with TypeScript and Python code. The directory structure is like so
While analyzing the Python code, the CodeQL action fails with below error. Could you please help me figure out what I might be doing wrong?
codeql.yml
codeql-config.yml
The text was updated successfully, but these errors were encountered: