C# Low Quality Scan

[rzuckerm]

Description of the issue

In TheRenegadeCoder/sample-programs, I'm getting the following warning from CodeQL:

Scanning C# code completed successfully, but the scan encountered issues. This may be caused by problems identifying dependencies or use of generated source code, among other reasons -- see other CodeQL diagnostics reported on the CodeQL status page for more details of possible causes. Addressing these warnings is advisable to avoid false-positive or missing results. If they cannot be addressed, consider scanning C# using either the autobuild or manual build modes.

I am build in manual mode, so I'm at a loss as to what CodeQL is complaining about. See this GitHub action run for details.

Note that this repo is unique in that there are no C# project files. Each C# file is intended to be a stand-alone file that has no external library dependencies.

[rvermeulen]

Hi @rzuckerm,

Thanks for you question. That is indeed confusing. Let me look into the warning.

[rvermeulen]

Hi @rzuckerm,

Minor update. This warning happens when the quality of the database is determined to be low which can happen when build-mode none is used, but for some unexpected reason happens now as well.

Will further investigate.

[rvermeulen]

Hi @rzuckerm,

We had a look at the workflow that you are using.
Based on https://github.com/TheRenegadeCoder/sample-programs/blob/e97127a8ce9b3ad30625032664d565a4df8b07df/scripts/get_codeql_languages.py#L66 and https://github.com/TheRenegadeCoder/sample-programs/blob/e97127a8ce9b3ad30625032664d565a4df8b07df/.github/workflows/codeql-analysis.yml#L123 it seems that you remove any unchanged files from the set of files being analyzed. Is that correct?

If so, that would impact the analysis. CodeQL is expected to analyze the entire source base of a project to be able to do inter procedural analysis.

[rzuckerm]

@rvermeulen Yes, that is correct. In order to streamline the pull request process and reduce the amount of time that a submitter has to wait for the GitHub Actions to complete, anything that hasn't changed is removed. What do you recommend?

[rzuckerm]

Another purpose is to reduce confusion so that a submitter doesn't see a failure for something they didn't change. A lot of our contributors are students with little or no experience using GitHub, so this helps them out as well.

Keep in mind that the purpose of this repo is to provide stand-alone code samples for a set of pre-determined projects in many
different programming languages. The code samples have no external dependencies. That's one of the rules for contributing.

[rvermeulen]

Unfortunately CodeQL won't work well with partial code bases and that is why it emits the warning.

In our PR process CodeQL scans the entire codebase, but only surfaces alerts on locations changed. I'm not sure if that process can be used with your students, but that is the experience we support.

The problem with filtering paths or alerts is that this creates an alert wobble. Code Scanning assumes that if an alert is no longer present in a scan, that it is fixed.

[rzuckerm]

@rvermeulen Thanks for your quick reply. I'll run this by our other maintainer, and see what they think.

@jrg94 FYI

[rzuckerm]

@rvermeulen It looks like you're right about partial runs. We also have a weekly schedule that runs the CodeQL scan on all the languages that support scanning, and C# is not seeing this issues. I'll discuss this tonight with the other maintainer. I think the way we're going to have to handle this is to scan an entire language when anyone does a PR for that language

[rzuckerm]

@rvermeulen I have a PR for a change to our CodeQL workflow where no files are removed, and I'm still seeing the same warning.

[rvermeulen]

Are the source files build in a container https://github.com/TheRenegadeCoder/sample-programs/blob/e97127a8ce9b3ad30625032664d565a4df8b07df/archive/c/c-sharp/testinfo.yml#L5?

[rzuckerm]

No, everything is running outside of docker for the CodeQL scan. For the CodeQL workflow, the testinfo.yml is used for the build command for the different languages that we run the scan on. Docker is only used to test the different code samples against the project requirements.

[rvermeulen]

Thanks for confirming, I will discuss with the C# team the conditions for surfacing this warning to see if that leads us to better understand why we are seeing it.

[rzuckerm]

It should be noted that we use CodeQL to scan all the supported languages, and C# is the only one we're having problem with