CODEQL - Scan passed even if alert introduced in PR, new alert can be found in Code Scanning tab

[Treders]

Hello,

I'm struggling with issue with CodeQL.

Setup:
Fork of the repository.
CodeQL Advanced enabled with codeql.yml

Steps:

1. Add custom query.
2. Prepare PR with new file with vulnerability.
3. Push the PR
4. Scan started, ongoing and then passed (even if there is new vulnerability introduced)
5. Check details of scan, it states "No new alerts found"
6. Go to "Code scanning" tab
7. New vulnerability is visible on that tab but for PR is not visible. By that it's not blocking PR

Has some of you encounter this problem?
Setup seems to work, it's finding problems but not showing them on PR, but in "Code scanning" tab.

[sampart]

Hi Adam, thanks for getting in touch. Would you be happy sharing the URL of your fork to help us dig into what's going on here? Thanks

(PS I'm away all next week so don't be surprised if you don't hear from me, or if you hear from one of my colleagues instead.)

[mdp2014]

salut la commu

[Treders]

Hi, unfortunatelly it's not my private repository so can't do it
My workflow looks like default one with
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v3
with:
category: "/language:go"

at the end, sarif file is uploaded but there is no info if some problem found and job in Github Action is passing

[sampart]

Hi again Adam,

Talking to my GitHub colleagues, one point they reminded me of is that alerts in Pull Requests only show up on the PR if they're present in the diff for the PR. Otherwise they only show up on the Code Scanning page as you mentioned.

Could this explain your situation?

If this is not the case and you are not able to share the repo name here so we can investigate further, then you've got a couple of options: You could either try to reproduce in a repo you are willing to share or have the repo owner open a support ticket which will keep information confidential between them and us. Thanks

[mdp2014]

salu la comu

[Treders]

Hi, thanks for suggestion.
I found out that when I introduced vulnerability from "default" pack it is shown correctly in PR, but when it should find vulnerability from custom_query, it's not finding it.

Here is my part of code for it:
yml file:

queries: +./.github/codeql/codeql-custom-queries-go # Include custom queries

custom query:

/**
 * @name Missing Copyright in .go file
 * @description Finds and lists all Go files (.go) that do not contain the string "Copyrights" in their comments.
 * @kind problem
 * @problem.severity error
 * @security-severity 5.0
 * @id go/find-missing-copyrights
 */

 import go

 /**
  * This query identifies Go files (.go) that do not contain the string "Copyrights" in comments.
  */
 
 // Check if a file contains the "Copyrights" string in comments
 predicate containsCopyrights(File f) {
   exists(Comment c |
     c.getFile() = f and
     c.getText().matches("%Copyright%")
   )
 }
 
 // Select Go files without the specified string in comments
 from File f
 where f.getBaseName().regexpMatch(".+\\.go$")
      and not containsCopyrights(f)
 select f, "Missing string: Copyrights in comments"

[sampart]

Thanks for the info. Investigating this without knowing which repo it is is difficult. If you're not happy to share the repo name here so we can investigate further, then you've got a couple of options: You could either try to reproduce in a repo you are willing to share or have the repo owner open a support ticket which will keep information confidential between them and us. Up to you which to do next. Thanks

[Treders]

Maybe let's do it in other way. Could you share some example repository link where there are custom queries introduced and when new vulnerability is introduced in PR it shows alert as expected?

Based on that I will compare existing solution and adjust if needed, maybe there are some missing parameters/values in my setup

[sampart]

I'm having a look to see if we know of an example repository of that sort. In the mean time, a couple of questions.

You gave me this extract from your YAML file:

queries: +./.github/codeql/codeql-custom-queries-go # Include custom queries

1. Just to confirm - this is from your Actions workflow file, not a custom configuration file, right?
2. Are you using a custom configuration file as well? The + syntax is for combining query configuration in the workflow with that in the custom config file, hence asking.
3. The line above doesn't include the default queries. If they're not included in a custom configuration file then you might want to include them here to ensure that they run. That would look like this (adding default,):

queries: +default,./.github/codeql/codeql-custom-queries-go # Include custom queries

[Treders]

Hi,

1. Yes
2. I have only one yml file with the workflow
3. Default scope of tests is running, I see it in logs

[sampart]

Could you share some example repository link where there are custom queries introduced and when new vulnerability is introduced in PR it shows alert as expected

This ask is quite specific and so we didn't have an example to hand, I'm afraid.

I'm going to repeat what I said before: Investigating this without knowing which repo it is is difficult. If you're not happy to share the repo name here so we can investigate further, then you've got a couple of options: You could either try to reproduce in a repo you are willing to share or have the repo owner open a support ticket which will keep information confidential between them and us.

We're limited in how much more support we can give in this discussion without you taking one of the approaches above, so over to you which to choose. Thanks

[sampart]

In tandem with the discussion above, here's another troubleshooting step you might like to try, courtesy of one of my colleagues:

If the user is not seeing an alert when they think they should, they should look at the logs and make sure the query is actually being run. In the analyze step, there is a log entry Running queries for ${language}. It lists all the queries that are run.

Here's an example of that output:

Hope this helps. Thanks

[Treders]

I see scans in logs. When I will add to PR new file with vulnerability, after scan I see new vulnerability in Code Scanning tab but not in the PR. It's scanning correctly but not displaying it in the PR (for custom query result)