Not able to use config file for database creation command

[HarshadDGhorpade-eaton]

** Our build process comprises of 4-5 commands so trying to use config file and use it in command but getting error as "Invalid property specified in the configuration file. Ignoring it and proceeding" **

as per using-a-codeql-configuration-file, does config file gets used internally by codeql database create without specifying --codescanning-config option ? Some internet sources talks about yaml-based config file as well and using it with --codescanning-config option, can you please clarify what's correct way to use config file?

I am trying this way :

codeql database create --language=cpp --github-url=https://github.com/ --codescanning-config=../codeql-config.yml --source-root . db

where codeql-config.yml file contents are like below :

name: My CodeQL Configuration
language: cpp
build:
  - "./setup.sh <arguments>"
  - "bash -c "command""
  - "./setup2.sh"<arguments>"
  - "bash -c "command""

getting below error :

Invalid property specified in the configuration file. Ignoring it and proceeding.
A fatal error occurred: Query pack codeql/cpp-queries cannot be found. Check the spelling of the pack.

Specifying multiple commands works but that becomes not maintainable as commands are lengthy :
codeql database create --command "cmd1" --command "cmd2" --command "cmd3" --command "cmd4" --language=cpp --github-url=https://github.com/ --source-root . db

[aibaars]

Have you tried using command: as the property name instead of build: ?

[HarshadDGhorpade-eaton]

Have you tried using command: as the property name instead of build: ?

still the same error with command.

[aibaars]

Could you try removing the name: property?

[HarshadDGhorpade-eaton]

Could you try removing the name: property?

Doesnt work, tried keeping only command/build, still same.

[aibaars]

Sorry, I think the documentation for the codescanning config file is the following https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/customizing-code-scanning#using-a-custom-configuration-file . I don't see any mention there for a command or build property.

The documentation you were referring to before can be used to provide default values for command line arguments.

I think neither is really suitable for your use-case. The easiest is probably to put the commands in a single shell script (for example build.sh) and run codeql database create --language cpp --command ./build.sh ....

Note that CodeQL may automatically recognize build.sh as a build script, so things may even work if you leave out --command ./build.sh.

[HarshadDGhorpade-eaton]

Sorry, I think the documentation for the codescanning config file is the following https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/customizing-code-scanning#using-a-custom-configuration-file . I don't see any mention there for a command or build property.

The documentation you were referring to before can be used to provide default values for command line arguments.

I think neither is really suitable for your use-case. The easiest is probably to put the commands in a single shell script (for example build.sh) and run codeql database create --language cpp --command ./build.sh ....

Note that CodeQL may automatically recognize build.sh as a build script, so things may even work if you leave out --command ./build.sh.

Thanks alot for this, it worked this way, sadly it's not mentioned in the doc anywhere but your replies were faster that solved issue quickly.

[HarshadDGhorpade-eaton]

while trying this out we're facing another issue :

ERROR: ld.so: object '/mnt/work/codeql/tools/linux64/lib64trace.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
......
ERROR: ld.so: object '/mnt/work/codeql/tools/linux64/lib64trace.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.

not getting what's issue here, this is causing failures in codeql database create command step in azure pipeline.
is this error harmless as written here ? not exactly the same error though.

I tried proceeding further with codeql database analyze but its saying "generated db needs to be finalized before running queries; please run codeql database finalize"

do I need to add codeql database finalize ?

[aibaars]

while trying this out we're facing another issue :

ERROR: ld.so: object '/mnt/work/codeql/tools/linux64/lib64trace.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
......
ERROR: ld.so: object '/mnt/work/codeql/tools/linux64/lib64trace.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.

not getting what's issue here, this is causing failures in codeql database create command step in azure pipeline. is this error harmless as written here ? not exactly the same error though.

Those error could be harmless if they only happen on processes the CodeQL analyser does not care about. In the linked example of a similar error, CodeQL was running inside a docker container and the error was reported on things running on the host machine.

In your case the errors suggest that CodeQL may be wrongly setting the LD_PRELOAD variable to the 64bit library for 32 bit processes. CodeQL needs to be able to "see" all compiler processes to figure out how to analyse the source code. So if the 32bit processes that are ignored happen to be the build scripts and compiler processes then CodeQL won't "see" a thing, and you end up with an empty database.

I'm a little surprised by the LD_PRELOAD value, I think it normally looks like /mnt/work/codeql/tools/linux64/${LIB}trace.so, and ld.so expands ${LIB} to the suitable value depending on whether the process is 32 or 64 bits. Could you validate that the LIB placeholder is not accidentally interpreted/replaced by your azure pipeline scripts?

I tried proceeding further with codeql database analyze but its saying "generated db needs to be finalized before running queries; please run codeql database finalize"

do I need to add codeql database finalize ?

No, normally codeql database create will run codeql database finalize automatically, except when database creation failed in an earlier step. You could try running codeql database finalize but even if it doesn't fail completely, you still end up with a partial database.

[HarshadDGhorpade-eaton]

Could you validate that the LIB placeholder is not accidentally interpreted/replaced by your azure pipeline scripts?

I am not sure what and how to check that, can you please elaborate on this ?

[aibaars]

Could you validate that the LIB placeholder is not accidentally interpreted/replaced by your azure pipeline scripts?

I am not sure what and how to check that, can you please elaborate on this ?

Could you try running printenv (or another command that prints the environment such as set or export) in your build script and look for the value of LD_PRELOAD?

Does your azure pipeline run a simple codeql database create command, or does it try to do more fancy things by setting special environment variables or using features like indirect build tracing ? If you're using a simple codeql database create then things should just work.

Could you also check which operating system and version is running on the azure devops workers? Do they run in docker or is some kind of virtualization or WSL in use? Perhaps running in a container may somehow confuse the code that detects whether a binary is 32 or 64bit .

[HarshadDGhorpade-eaton]

I dont see LD_PRELOAD environment variable after doing. Earlier I was printing env vars after the db creation command in a separate step so getting nothing, when printing same in build.sh, I do find LD_PRELOAD=/mnt/work/codeql/tools/linux64/lib64trace.so

database creation command :
codeql database create --language cpp --github-url=https://github.com/ --command ./build.sh --source-root . db

OS details :
Distributor ID: Ubuntu
Description: Ubuntu 20.04.6 LTS
Release: 20.04
x86_64

We're running on Azure cloud agents, nothing on container.

[HarshadDGhorpade-eaton]

database-create-20230622.152801.058.log
Uploading database create log just in case if it helps.

[aibaars]

separate step so getting nothing, when printing same in build.sh, I do find LD_PRELOAD=/mnt/work/codeql/tools/linux64/lib64trace.so

Thanks! Could you attach a list of all environment variables containing any of the words PRELOAD, SEMMLE, ODASA, and CODEQL?

Could you also attach the build-tracer.log file to this issue?

[HarshadDGhorpade-eaton]

Was trying to attach build-tracer.log but its more than 2GB.. zipped version goes ~180MB.. can attached only until 25 MB here.

PRELOAD -->
SEMMLE_PRELOAD_libtrace=/mnt/work/codeql/tools/linux64/${LIB}_${PLATFORM}_trace.so
SEMMLE_PRELOAD_libtrace32=/mnt/work/codeql/tools/linux64/lib32trace.so
SEMMLE_PRELOAD_libtrace64=/mnt/work/codeql/tools/linux64/lib64trace.so
LD_PRELOAD=/mnt/work/codeql/tools/linux64/lib64trace.so

SEMMLE -->
SEMMLE_PRELOAD_libtrace=/mnt/work/codeql/tools/linux64/${LIB}_${PLATFORM}_trace.so
SEMMLE_PRELOAD_libtrace32=/mnt/work/codeql/tools/linux64/lib32trace.so
SEMMLE_PRELOAD_libtrace64=/mnt/work/codeql/tools/linux64/lib64trace.so
SEMMLE_EXEC=

ODASA --> nothing

CODEQL -->
CODEQL_EXTRACTOR_CPP_TRAP_DIR=/mnt/work/1/s/edge-linux-yocto/yocto_db/trap/cpp
CODEQL_TRACER_DIAGNOSTICS_DIR=/mnt/work/1/s/edge-linux-yocto/yocto_db/diagnostic/tracer
CODEQL_EXTRACTOR_CPP_LOG_DIR=/mnt/work/1/s/edge-linux-yocto/yocto_db/log
CODEQL_EXTRACTOR_CPP_SOURCE_ARCHIVE_DIR=/mnt/work/1/s/edge-linux-yocto/yocto_db/src
CODEQL_PLATFORM_DLL_EXTENSION=.so
CODEQL_EXTRACTOR_CPP_DIAGNOSTIC_DIR=/mnt/work/1/s/edge-linux-yocto/yocto_db/diagnostic/extractors/cpp
CODEQL_EXTRACTOR_CPP_WIP_DATABASE=/mnt/work/1/s/edge-linux-yocto/yocto_db
CODEQL_JAVA_HOME=/mnt/work/codeql/tools/linux64/java
CODEQL_EXTRACTOR_CPP_SCRATCH_DIR=/mnt/work/1/s/edge-linux-yocto/yocto_db/working
CODEQL_DIST=/mnt/work/codeql
CODEQL_PLATFORM=linux64
CODEQL_SCRATCH_DIR=/mnt/work/1/s/edge-linux-yocto/yocto_db/working
CODEQL_TRACER_LANGUAGES=cpp
CODEQL_TRACER_LOG=/mnt/work/1/s/edge-linux-yocto/yocto_db/log/build-tracer.log
CODEQL_EXTRACTOR_CPP_ROOT=/mnt/work/codeql/cpp
CODEQL_PARENT_ID=0000000000001561_0000000000000003
CODEQL_EXEC_ARGS_OFFSET=