Problem with installing additional dependencies

[mutasDev]

Hey, im trying to run my own query set containing a .ql query for python IncompleteHostNameRegExp taken from the official CodeQL Repository.

When I try to compile the queries, ten of them compile and can run through, but the IncompleteHostNameRegExp fails with the following error:

ERROR: Could not resolve module semmle.python.security.regexp.HostnameRegex (/Users/mutas/Repositories/master-thesis/Analysis/Queries/py/top25/top25/codeql/IncompleteHostnameRegExp.ql:14,16-59)
ERROR: Could not resolve module HostnameRegex (/Users/mutas/Repositories/master-thesis/Analysis/Queries/py/top25/top25/codeql/IncompleteHostnameRegExp.ql:16,28-41)
ERROR: Query module does not have any queries in scope. (/Users/mutas/Repositories/master-thesis/Analysis/Queries/py/top25/top25/codeql/IncompleteHostnameRegExp.ql:1,1-1)

The ql. file contains the following content:

private import semmle.python.security.regexp.HostnameRegex as HostnameRegex

query predicate problems = HostnameRegex::incompleteHostnameRegExp/4;

1. Is there any special way I have to include the semmle library?
2. I have created the ql. set and then added some more queries afterwards, is that a problem?

I have tried to reinstall the python-queries dependencies, but when I try to do that I get the following error, too:

codeql pack add codeql/python-queries
Adding codeql/python-queries@latest to dependencies.
Saving qlpack.yml.
Reinstalling dependencies.
A fatal error occurred: Package version failure. No compatible version found: codeql/suite-helpers@0.4.1.

[aeisenberg]

The problem that you are facing is that the query pack you are trying to run depends on codeql/suite-helpers@0.4.1 (possibly a transitive dependency). I'm not entirely sure what context you are running your queries in and if they are in a query pack right now (look for a qlpack.yml or codeql-pack.yml file in the directory or a parent).

If there is no file, create one and add the minimal content:

name: hucairz
dependencies:
  - codeql/python-all: "*"

In the directory containing this file, run the command:

codeql pack install

This will download and install all dependencies of your pack, including transitive dependencies. Then you can re-run your queries.

More information about codeql packs can be found in the documentation. https://codeql.github.com/docs/codeql-cli/about-codeql-packs/

[aeisenberg]

OK. I understand now. The error you are seeing is happening during the "interpretation" step. Running a query will initially produce a bqrs file (essentially a binary file with the results in it). It then needs to be "interpreted" into a format that is human readable. Certain formats require certain kinds of query metadata. From the look of your error, I am guessing that you are trying to get the query results into a sarif format?

If so, you have two possibilities:

1. Create the query metadata appropriate for this format (off the top of my head, I think you need @kind problem). You may also need to change the select clause. Read the docs for more info.
2. Change the output format to something simpler, like csv.

[mutasDev]

I am already trying to export the results as csv and it is still not working. I have changed the kind to problem too, but I dont think I am experienced enough to rewrite the query!

Thanks for your help tho, in the worst case I will just omit that query from my set !

[mutasDev]

Another short remark/Question: The query is from the official Security-Directory of the python codeql queries in this repository.
Is there maybe a scenario that my pack is missing something that the official query sets include and therefore the interpretation does not work?

[aeisenberg]

Apologies, I gave you incorrect advice above. Queries of type table cannot be interpreted and they can only be interacted with in the bqrs format. This particular query, even though it is in our default package, is ignored since it is a table query. The default suite of all packs ignore table queries and only include problem, path-problem, alert, path-alert, metric, and diagnostic queries. See this query suite for how it works: https://github.com/github/codeql/blob/main/misc/suite-helpers/code-scanning-selectors.yml

So, if you are trying to mimic all the queries in the default query suite of the python pack, it is ok to ignore table queries.

[mutasDev]

That clarifies it! Thanks for all the insights and help throughout the days!