Python extractor failure when Python 3.6 is used

[bullhacks3]

Issue:

I'm testing custom queries (simple) in python using codeql 2.15.3 version, but stucked in the testing phase itself.

Python version = 3.11

One more issue: github/codeql-action#2042

Note: Currently I'm using default setup no customization, still issue is happening only for python (for other languages such javascript, java, I'm able to run query without any issues). Please have a look .

CodeQL Version

└──╼ $/home/bagupta/codeql/codeql --version
CodeQL command-line toolchain release 2.15.3.
Copyright (C) 2019-2023 GitHub, Inc.
Unpacked in: /home/bagupta/codeql
Analysis results depend critically on separately distributed query and
extractor modules. To list modules that are visible to the toolchain,
use 'codeql resolve qlpacks' and 'codeql resolve languages'.

Command:
/home/bakul/codeql/codeql test run UrlRedirect.ql --show-extractor-output

Output:

Executing 1 tests in 1 directories.
Extracting test database in /home/bakul/codeql/python/ql/src/experimental/security/url-redirect.
[2024-01-16 00:29:46] [build-err] Process ForkProcess-1:
[2024-01-16 00:29:46] [build-err] Traceback (most recent call last):
[2024-01-16 00:29:46] [build-err] File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
[2024-01-16 00:29:46] [build-err] self.run()
[2024-01-16 00:29:46] [build-err] File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
[2024-01-16 00:29:46] [build-err] self._target(self._args, self._kwargs)
[2024-01-16 00:29:46] [build-err] File "/home/bakul/codeql/python/tools/python3src.zip/semmle/logging.py", line 116, in _message_loop
[2024-01-16 00:29:46] [build-err] sys.stdout.reconfigure(encoding='utf-8')
[2024-01-16 00:29:46] [build-err] AttributeError: '_io.TextIOWrapper' object has no attribute 'reconfigure'
[2024-01-16 00:29:46] [build-err] Traceback (most recent call last):
[2024-01-16 00:29:46] [build-err] File "/home/bakul/codeql/python/tools/python_tracer.py", line 53, in
[2024-01-16 00:29:46] [build-err] semmle.populator.main(original_path)
[2024-01-16 00:29:46] [build-err] File "/home/bakul/codeql/python/tools/python3src.zip/semmle/populator.py", line 43, in main
[2024-01-16 00:29:46] [build-err] AttributeError: '_io.TextIOWrapper' object has no attribute 'reconfigure'
[2024-01-16 00:29:46] [ERROR] Spawned process exited abnormally (code 1; tried to run: [python3, /home/bakul/codeql/python/tools/python_tracer.py, --lang=3, --filter=exclude:/.testproj/**, --path, /home/bakul/codeql/python/ql/src/experimental/security/url-redirect, --verbosity, 3, --colorize])
Could not extract a dataset in /home/bakul/codeql/python/ql/src/experimental/security/url-redirect: Extraction command python3 failed with status 1.
Extraction command python3 failed with status 1.
[1/1] FAILED(EXTRACTION) /home/bakul/codeql/python/ql/src/experimental/security/url-redirect/UrlRedirect.ql
Compiling queries in /home/bakul/codeql/python/ql/src/experimental/security/url-redirect.
Completed in 4.5s (extract 1.2s comp 0ms eval 0ms).
0 tests passed; 1 tests failed:
FAILED: /home/bakul/codeql/python/ql/src/experimental/security/url-redirect/UrlRedirect.ql

[RasmusWL]

as already mentioned elsewhere:

A web search for AttributeError: '_io.TextIOWrapper' object has no attribute 'reconfigure' suggests this problem can be solved by using Python version 3.7 or higher. It looks like your self-hosted runner has python 3.6 which is pretty old. Could you try upgrading the python version?

---

[2024-01-16 00:29:46] [build-err] File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap

This line in the output suggests that Python 3.6 is being used. Can you please try the following commands and provide the output?

1. python --version
2. python3 --version
3. bash -c 'python --version'
4. if those don't reveal anything, can you run your test with --show-extractor-output and show the logs? (it should contain a line saying what python version is being used)

[bullhacks3]

Thanks @RasmusWL , @sidshank . Yupp while building the code, it was using system installed python environment (3.6). Now I've changed it to explicitly use 3.7 . Thanks, able to resolve this query. Thanks for the quick response, and apology for delay from my end.