General issue: Cannot upgrade database

[JeffJerseyCow]

I've been following the guidance on how to configure CodeQL, create a database and analyze with a query pack. But I always end up with the following error:

A fatal error occurred: This CodeQL database is not compatible with the QL library /home/chris/projects/codeql/codeql-repo/python/ql/src/Security/CWE-502/UnsafeDeserialization.ql uses. You may need to run 'codeql database upgrade testDb'.

It doesn't matter what version of CodeQL I'm using, how many times I update the database or the queries I use. It's always the same error.

It's the case for the CPP, JavaScript and Python queries.

My directory that holes everything CodeQL is here.

$pwd
/home/chris/projects/codeql

The CodeQL version is 2.2.4 (I've also tried it with earlier versions) and is moved to a codeql-cli directory.

$curl -LO https://github.com/github/codeql-cli-binaries/releases/download/v2.2.4/codeql-linux64.zip
$unzip codeql-linux64.zip 
$mv codeql codeql-cli

I use the latest queries extracted to codeql-repo.

$ git clone https://github.com/github/codeql.git codeql-repo

I create the database and put it in dukDatabase.

codeql database create dukDatabase --language=cpp --command=make --source-root=../duktape

Then I try to analyse with a query, or query pack and it always fails.

codeql database analyze dukDatabase/ codeql-repo/cpp/ql/src/codeql-suites/cpp-lgtm.qls --format=sarifv2 --output=duk.sarifv2 --search-path=codeql-repo/

I always get an error complaining about not finding dependencies, but I add --search-path to resolve it. But I always get the error about an incompatible database.

I've followed the guidance to update the database, it just says it's already updated.

It doesn't matter what version of codeql I used or queries...it never works.

Any help is appreciated.

[RasmusWL]

Hi @JeffJerseyCow , sorry to hear you're having trouble with upgrading the database 😐

I reproduced your setup with codeql-cli and codeql-repo folders, but was not able to reproduce your upgrading problems.

Can you create a dummy DB like this

$ mkdir src-test
$ echo "print(42)" > src-test/wat.py
$ ./codeql-cli/codeql database create --language python --source-root src-test/ test-db

and show the output from running

$ ./codeql-cli/codeql database upgrade -vv test-db

[carbureted]

I'm having the same issue (also with a C++ codebase) on OSX 10.14.6. Results of the above commands are below - looks like some dependencies might be missing?

codeql database create --language python --source-root src-test/ test-db
Initializing database at ./test-db.
Running command [codeql/python/tools/autobuild.sh] in ./src-test.
[2020-08-19 16:26:44] [build-err] ERROR:root:code for hash md5 was not found.
[2020-08-19 16:26:44] [build-err] Traceback (most recent call last):
[2020-08-19 16:26:44] [build-err]   File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
[2020-08-19 16:26:44] [build-err]     globals()[__func_name] = __get_hash(__func_name)
[2020-08-19 16:26:44] [build-err]   File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
[2020-08-19 16:26:44] [build-err]     raise ValueError('unsupported hash type ' + name)
[2020-08-19 16:26:44] [build-err] ValueError: unsupported hash type md5
[2020-08-19 16:26:44] [build-err] ERROR:root:code for hash sha1 was not found.
[2020-08-19 16:26:44] [build-err] Traceback (most recent call last):
[2020-08-19 16:26:44] [build-err]   File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
[2020-08-19 16:26:44] [build-err]     globals()[__func_name] = __get_hash(__func_name)
[2020-08-19 16:26:44] [build-err]   File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
[2020-08-19 16:26:44] [build-err]     raise ValueError('unsupported hash type ' + name)
[2020-08-19 16:26:44] [build-err] ValueError: unsupported hash type sha1
[2020-08-19 16:26:44] [build-err] ERROR:root:code for hash sha224 was not found.
[2020-08-19 16:26:44] [build-err] Traceback (most recent call last):
[2020-08-19 16:26:44] [build-err]   File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
[2020-08-19 16:26:44] [build-err]     globals()[__func_name] = __get_hash(__func_name)
[2020-08-19 16:26:44] [build-err]   File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
[2020-08-19 16:26:44] [build-err]     raise ValueError('unsupported hash type ' + name)
[2020-08-19 16:26:44] [build-err] ValueError: unsupported hash type sha224
[2020-08-19 16:26:44] [build-err] ERROR:root:code for hash sha256 was not found.
[2020-08-19 16:26:44] [build-err] Traceback (most recent call last):
[2020-08-19 16:26:44] [build-err]   File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
[2020-08-19 16:26:44] [build-err]     globals()[__func_name] = __get_hash(__func_name)
[2020-08-19 16:26:44] [build-err]   File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
[2020-08-19 16:26:44] [build-err]     raise ValueError('unsupported hash type ' + name)
[2020-08-19 16:26:44] [build-err] ValueError: unsupported hash type sha256
[2020-08-19 16:26:44] [build-err] ERROR:root:code for hash sha384 was not found.
[2020-08-19 16:26:44] [build-err] Traceback (most recent call last):
[2020-08-19 16:26:44] [build-err]   File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
[2020-08-19 16:26:44] [build-err]     globals()[__func_name] = __get_hash(__func_name)
[2020-08-19 16:26:44] [build-err]   File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
[2020-08-19 16:26:44] [build-err]     raise ValueError('unsupported hash type ' + name)
[2020-08-19 16:26:44] [build-err] ValueError: unsupported hash type sha384
[2020-08-19 16:26:44] [build-err] ERROR:root:code for hash sha512 was not found.
[2020-08-19 16:26:44] [build-err] Traceback (most recent call last):
[2020-08-19 16:26:44] [build-err]   File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
[2020-08-19 16:26:44] [build-err]     globals()[__func_name] = __get_hash(__func_name)
[2020-08-19 16:26:44] [build-err]   File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
[2020-08-19 16:26:44] [build-err]     raise ValueError('unsupported hash type ' + name)
[2020-08-19 16:26:44] [build-err] ValueError: unsupported hash type sha512
[2020-08-19 16:26:44] [build] Calling python3 python/tools/get_venv_lib.py
[2020-08-19 16:26:44] [build] Calling python3 -S python/tools/python_tracer.py -v -z all -c ./test-db/working/trap_cache -p /usr/local/lib/python3.8/site-packages -R ./src-test
[2020-08-19 16:26:45] [build] [INFO] Python version 3.8.5
[2020-08-19 16:26:45] [build] [INFO] Python extractor version 5.11
[2020-08-19 16:26:46] [build] [INFO] [4] Extracted file ./src-test/wat.py in 9ms
[2020-08-19 16:26:46] [build] [INFO] Processed 1 modules in 1.04s

[RasmusWL]

Hi @carbureted , although those are error messages, it doesn't look critical, and it seems like the database was created succesfully. What problems are you seeing when running queries against it? (such as ./codeql-cli/codeql query run --database test-db codeql-repo/python/ql/examples/snippets/backticks.ql)