[cpp] database create problems

[lianxv-primer]

I use command creating databases , like :
codeql database create C:\test\codeql-database --source-root "E:\test-project-code\src" --language=cpp --command="call build_win_codeql.bat" --threads=0 --verbose --overwrite --mode=clear --min-disk-free=100000

it built project successful,
but it takes too much time and costs too much disks . and after build it costs much more time to scanning trap file.
is it something wrong

all build-trace like :

...
...
[2024-04-16 11:42:00] [build-stdout] MSBuild version 17.9.8+b34f75857 for .NET Framework
[2024-04-16 11:42:00] [build-stdout] Build started 4/16/2024 11:42:00 AM.
...
[2024-04-16 21:04:28] [build-stdout] 4134 Warning(s)
[2024-04-16 21:04:28] [build-stdout] 0 Error(s)
[2024-04-16 21:04:28] [build-stdout] Time Elapsed 09:22:26.82
[2024-04-16 21:04:28] Plumbing command codeql database trace-command completed.
[2024-04-16 21:04:28] [PROGRESS] database create> Finalizing database at C:\test\devops-codeql-database.
[2024-04-16 21:04:28] Running plumbing command: codeql database finalize --threads=0 --mode=clear --min-disk-free=10000 --no-db-cluster -- C:\test\devops-codeql-database
[2024-04-16 21:04:28] Using pre-finalize script C:\codeql-home\codeql\cpp\tools\pre-finalize.cmd.
[2024-04-16 21:04:28] [PROGRESS] database finalize> Running pre-finalize script C:\codeql-home\codeql\cpp\tools\pre-finalize.cmd in C:\devops\p-6ac70a2931f74eb2a2452fb8f52372e1\src.
[2024-04-16 21:04:28] Running plumbing command: codeql database trace-command --working-dir=C:\devops\p-6ac70a2931f74eb2a2452fb8f52372e1\src --no-tracing --threads=0 -- C:\test\devops-codeql-database C:\codeql-home\codeql\cpp\tools\pre-finalize.cmd
[2024-04-16 21:04:28] [PROGRESS] database trace-command> Running command in C:\devops\p-6ac70a2931f74eb2a2452fb8f52372e1\src: [C:\codeql-home\codeql\cpp\tools\pre-finalize.cmd]
[2024-04-16 21:04:28] Plumbing command codeql database trace-command completed.
[2024-04-16 21:04:28] [PROGRESS] database finalize> Running TRAP import for CodeQL database at C:\test\devops-codeql-database...
[2024-04-16 21:04:28] Running plumbing command: codeql dataset import --dbscheme=C:\codeql-home\codeql\cpp\semmlecode.cpp.dbscheme --threads=0 -- C:\test\devops-codeql-database\db-cpp C:\test\devops-codeql-database\trap\cpp
[2024-04-16 21:04:28] Clearing disk cache since the version file C:\test\devops-codeql-database\db-cpp\default\cache\version does not exist
[2024-04-16 21:04:29] Tuple pool not found. Clearing relations with cached strings
[2024-04-16 21:04:29] Trimming disk cache at C:\test\devops-codeql-database\db-cpp\default\cache in mode clear.
[2024-04-16 21:04:29] Sequence stamp origin is -6195435911585769745
[2024-04-16 21:04:29] Pausing evaluation to hard-clear memory at sequence stamp o+0
[2024-04-16 21:04:29] Unpausing evaluation
[2024-04-16 21:04:29] Pausing evaluation to quickly trim disk at sequence stamp o+1
[2024-04-16 21:04:29] Unpausing evaluation
[2024-04-16 21:04:29] Pausing evaluation to zealously trim disk at sequence stamp o+2
[2024-04-16 21:04:29] Unpausing evaluation
[2024-04-16 21:04:29] Trimming completed (12ms): Purged everything.
[2024-04-16 21:04:29] Scanning for files in C:\test\devops-codeql-database\trap\cpp
[2024-04-16 21:05:01] Found 18594 files on disk containing 7859950 TRAP files (239.02 GiB)
[2024-04-16 21:05:01] [PROGRESS] dataset import> Grouping TRAP files by link target
[2024-04-16 21:11:37] [PROGRESS] dataset import> Grouping unlinked TRAP files together
[2024-04-16 21:12:11] [PROGRESS] dataset import> Scanning TRAP files
...
[2024-04-16 21:55:32] Scanning trace_log.cc.11991bd7.trap (trace_log.cc.11991bd7_0.trap.tar.br) (6891731 of 7859950)
...
[2024-04-17 00:06:20] Importing 96f07e7c3376ac5f473f4bab.trap (trace_log.cc.11991bd7_0.trap.tar.br) for no link target (6974499 of 7859950) [2024-04-17 00:06:20] [ERROR] dataset import> 9fafff7e4bf067c0c66b0ca7.trap (connection.cc.a0da4be2_0.trap.tar.br) for no link target, 38: com.semmle.util.exception.CatastrophicError: ID 94380083 is already mapped to 19293495 com.semmle.inmemory.util.DiskIdStore.append(DiskIdStore.java:63) com.semmle.inmemory.util.NonSequentialDiskPool.insert(NonSequentialDiskPool.java:105) com.semmle.inmemory.util.NonSequentialDiskPool.insertBucket(NonSequentialDiskPool.java:47) com.semmle.inmemory.util.DiskPool.getIdWithFreshness(DiskPool.java:171) com.semmle.inmemory.util.NonSequentialDiskPool.getIdWithFreshness(NonSequentialDiskPool.java:14) com.semmle.inmemory.populate.IdPool.lookupWithFreshness(IdPool.java:66) com.semmle.inmemory.trap.SynchronizedIdAllocator.getElementIdAndFreshness(SynchronizedIdAllocator.java:23) com.semmle.inmemory.trap.FreshIdAllocator.getElementId(FreshIdAllocator.java:22) com.semmle.inmemory.trap.TRAPReader$MetaStringBuilder.getElementId(TRAPReader.java:1016) com.semmle.inmemory.trap.TRAPReader.computeID(TRAPReader.java:1090) com.semmle.inmemory.trap.TRAPReader.computeID(TRAPReader.java:1085) com.semmle.inmemory.trap.TRAPReader.scanLabelKey(TRAPReader.java:829) com.semmle.inmemory.trap.TRAPReader.scanLabelValue(TRAPReader.java:801) com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:505) com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:414) com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:234) com.semmle.inmemory.trap.ImportTasksProcessor.lambda$importTrap$1(ImportTasksProcessor.java:154) com.semmle.util.concurrent.FutureUtils.lambda$mapAsync_$8(FutureUtils.java:161) java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) java.base/java.lang.Thread.run(Unknown Source) at (failed to read line: asked for -1 bytes which seems wrong!)
...
...
(I stopped running because it always print this error messages in different trap files)

[jketema]

Thanks for opening a new issue for this. Could you tell me which version of CodeQL you are using? If this is not the latest version, could you try with the latest version, which should have better error in the area where the crash seems to occur.

[lianxv-primer]

Thanks for opening a new issue for this. Could you tell me which version of CodeQL you are using? If this is not the latest version, could you try with the latest version, which should have better error in the area where the crash seems to occur.

2.16.5

[jketema]

We've looked a bit closer at this and the following:

com.semmle.util.exception.CatastrophicError: ID 94380083 is already mapped to 19293495
...
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) java.base/java.lang.Thread.run(Unknown Source) at (failed to read line: asked for -1 bytes which seems wrong!)

combined with your remark that "(I stopped running because it always print this error messages in different trap files)" makes us suspect that the machine you're using to create the database might have bad RAM. Would it be possible for you to try using a different machine to create the database to see if the problem persists there?

[lianxv-primer]

We've looked a bit closer at this and the following:

com.semmle.util.exception.CatastrophicError: ID 94380083 is already mapped to 19293495
...
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) java.base/java.lang.Thread.run(Unknown Source) at (failed to read line: asked for -1 bytes which seems wrong!)

combined with your remark that "(I stopped running because it always print this error messages in different trap files)" makes us suspect that the machine you're using to create the database might have bad RAM. Would it be possible for you to try using a different machine to create the database to see if the problem persists there?

I got builds today in the same machine, It shows that "exausted all available IDs" here :
16: com.semmle.util.exception.CatastrophicError: We have exausted all available IDs in the disk pool C:\test\win-codeql-database\db-cpp\default\idPool
com.semmle.inmemory.util.DualSequentialDiskPool.checkAfterAllocatingId(DualSequentialDiskPool.java:94)
com.semmle.inmemory.util.DualSequentialDiskPool.getIdWithFreshness(DualSequentialDiskPool.java:77)
com.semmle.inmemory.populate.IdPool.lookupWithFreshness(IdPool.java:66)
com.semmle.inmemory.trap.SynchronizedIdAllocator.getElementIdAndFreshness(SynchronizedIdAllocator.java:23)
com.semmle.inmemory.trap.FreshIdAllocator.getElementId(FreshIdAllocator.java:22)
com.semmle.inmemory.trap.TRAPReader$MetaStringBuilder.getElementId(TRAPReader.java:1016)
com.semmle.inmemory.trap.TRAPReader.computeID(TRAPReader.java:1097)
com.semmle.inmemory.trap.TRAPReader.scanLabelKey(TRAPReader.java:829)
com.semmle.inmemory.trap.TRAPReader.scanLabelValue(TRAPReader.java:801)
com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:505)
com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:414)
com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:234)
com.semmle.inmemory.trap.ImportTasksProcessor.lambda$importTrap$1(ImportTasksProcessor.java:154)
com.semmle.util.concurrent.FutureUtils.lambda$mapAsync_$8(FutureUtils.java:161)
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base/java.lang.Thread.run(Unknown Source)

Maybe It is not RAM problem?

[jketema]

The "We have exausted all available IDs in the disk pool" message is what we expected. The other errors you were seeing should not have been possible because it should have reached this particular message. So we still suspect there is something wrong with your machine, but on top of that the tool cannot handle the size of the database you're trying to create. The latest version (if you were not using that already) might perform slightly better, but that might still not sufficient in your case.

If this is a big "mono repo" with many different applications in it, you might be better of creating a separate database per application. If this is not the case, then what we've discussed on #16237 is possibly your only option at the moment.

Note that we have been actively looking at the underlying causes of the "We have exausted all available IDs in the disk pool" message, and it is something we have been trying to improve. However, I cannot give you any time lines of when we might have something that would work in your case.

[lianxv-primer]

The "We have exausted all available IDs in the disk pool" message is what we expected. The other errors you were seeing should not have been possible because it should have reached this particular message. So we still suspect there is something wrong with your machine, but on top of that the tool cannot handle the size of the database you're trying to create. The latest version (if you were not using that already) might perform slightly better, but that might still not sufficient in your case.

Yes, I have updated to 2.17.0. And the latest log above is runned in 2.17.0. Maybe I have to reduce build scale, or maybe this is another way to reduce trace scale (for example : ignore trace some folders)? just to confirm

[jketema]

Maybe I have to reduce build scale, or maybe this is another way to reduce trace scale (for example : ignore trace some folders)? just to confirm

For compiled languages like C++, the tool does not support ignoring folders, so that's not an option. As I wrote above your options are basically:

• Build a separate database per application, if there is source code for multiple applications in your repo. This would require being able to build the applications in isolation, running codeql database create for each of them.
• Try the technique I described in [cpp] How to extract only part of files when creating database #16237

[lianxv-primer]

Maybe I have to reduce build scale, or maybe this is another way to reduce trace scale (for example : ignore trace some folders)? just to confirm

For compiled languages like C++, the tool does not support ignoring folders, so that's not an option. As I wrote above your options are basically:

• Build a separate database per application, if there is source code for multiple applications in your repo. This would require being able to build the applications in isolation, running codeql database create for each of them.
• Try the technique I described in [cpp] How to extract only part of files when creating database #16237

I get it , thank you

[lianxv-primer]

@jketema I divided into several parts. But It still have logs with "We have exausted all available IDs in the disk pool"。 And in the build-tracer.log I found many errors (6000+ times in build-tracer.log) like (num) errors detected in the compilation of (*.cpp):
"
3 errors detected in the compilation of "xxx.cpp".
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 21340] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 21340] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 21340] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 21340] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 21340] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 21340] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 21340] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 21340] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 21340] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 21340] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 21340] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 21340] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 21340] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_expr_node: Unknown expr kind 30.
[E 17:13:38 31400] Warning[extractor-c++]: In index_expr_node: Unknown expr kind 30.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 21340] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 21340] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 22972] Finished compilation TRAP C:\test\devops-codeql-database\trap\cpp/compilations/48/10416520_0.trap.br
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 22972] Marking C compilation as happened.
[E 17:13:38 22972] Setting tag c-compilation-happened
[E 17:13:38 22972] Starting from CODEQL_TRACER_DB_ID D1A20C3300007580_0000000000000001
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 22972] [E 17:13:38 31400] Locking DB C:\test\devops-codeql-database\working/tags.dbWarning[extractor-c++]: In
index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 22972] Locked DB C:\test\devops-codeql-database\working/tags.db
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 22972] Set tag for D1A20C3300007580_0000000000000001
[E 17:13:38 22972] Set tag for 1A7073F20000689C_0000000000000001
[E 17:13:38 22972] Set tag for DC4AE1650000091C_0000000000000001
[E 17:13:38 22972] Set tag for 770DCC5600008EC0_0000000000000001
[E 17:13:38 22972] Unlocking DB
[E 17:13:38 22972] Unlocked DB
[E 17:13:38 22972] Warning[extractor-c++]: In main: Extractor exiting with code 1
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_expr_node: Unknown expr kind 30.
[E 17:13:38 27220] Warning[extractor-c++]: In index_expr_node: Unknown expr kind 30.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 27220] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 18048] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 30768] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 30768] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 31400] Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
[E 17:13:38 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 20784] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 20784] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 3044] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:38 3044] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 3044] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 3044] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 31400] Warning[extractor-c++]: In index_expr_node: Unknown expr kind 30.
[E 17:13:39 31400] Warning[extractor-c++]: In index_expr_node: Unknown expr kind 30.
[E 17:13:39 27220] Warning[extractor-c++]: In index_expr_node: Unknown expr kind 30.
[E 17:13:39 18048] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 18048] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 21340] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 18048] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
[E 17:13:39 18048] Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
"
is it useful messages to help us find what happened here?

[jketema]

But It still have logs with "We have exausted all available IDs in the disk pool"

The only thing I can recommend is cut things up in even smaller parts, if possible.

All the warnings are pretty inconsequential, and the one that occurs most "In index_stmk_decl: No source sequence entry for declaration statement." is fixed in the latest CodeQL version.

To help you with "3 errors detected in the compilation of "xxx.cpp"." I would need to see the actually error messages (which should occur higher up in the log).

[lianxv-primer]

But It still have logs with "We have exausted all available IDs in the disk pool"

The only thing I can recommend is cut things up in even smaller parts, if possible.

All the warnings are pretty inconsequential, and the one that occurs most "In index_stmk_decl: No source sequence entry for declaration statement." is fixed in the latest CodeQL version.

To help you with "3 errors detected in the compilation of "xxx.cpp"." I would need to see the actually error messages (which should occur higher up in the log).

In fact, today I downgraded codeql to 2.15.4. Now the divided project can be compiled normally. The compiled database size is 13G and the time is within 6 hours (the intermediate files during the compilation process may be cleaned up at the end). In the before when upgrading to a newer version For versions such as (2.16.5 and 2.17.1 (released a few days ago)), compilation will take more than 12 hours and eventually fail.

[jketema]

In fact, today I downgraded codeql to 2.15.4. Now the divided project can be compiled normally. The compiled database size is 13G and the time is within 6 hours (the intermediate files during the compilation process may be cleaned up at the end).

Note that this might be because (i) 2.15.4 is able to extract significantly less code, so you will have less coverage, (ii) 2.1.5.4 doesn't check for exhausting of IDs in the disk pool, so you might silently end up with a corrupted database.

[lianxv-primer]

To help you with "3 errors detected in the compilation of "xxx.cpp"." I would need to see the actually error messages (which should occur higher up in the log).

The log occured in higher up is similar to what I just posted. I roughly categorized it (whole log is GB sizes):
Command: C:\codeql-home\codeql\cpp\tools\win64\extractor.exe --mimic @-
Already archived xxx.cpp/.h
Processed command line: C:\codeql-home\codeql\cpp\tools\win64\extractor.exe --trapfolder C:\test\devops-codeql-database\trap\cpp --src_archive ...
Not using TRAP cache
TRAP cache miss
Starting compilation TRAP C:\test\devops-codeql-database\trap\cpp/compilations/20/23230266_0.trap.br
Opening existencedb in \?\C:\test\devops-codeql-database\working\existencedb\db
Warning[extractor-c++]: In add_constructor_init: Unexpected dynamic init kind 7.
Warning[extractor-c++]: In index_stmk_decl: No source sequence entry for declaration statement.
Warning[extractor-c++]: In index_expr_node: Unknown expr kind 30.
In add_constructor_init: Unexpected dynamic init kind 1.
In add_constructor_init: Unexpected dynamic init kind 3.
...
Finished compilation TRAP C:\test\devops-codeql-database\trap\cpp/compilations/20/23230266_0.trap.br
Wrote 996 files to xxx.trap.tar.br
Marking C compilation as happened.
Setting tag c-compilation-happened
Starting from CODEQL_TRACER_DB_ID 6DCAA09F00007580_0000000000000001
Locking DB C:\test\devops-codeql-database\working/tags.db
Locked DB C:\test\devops-codeql-database\working/tags.db
Set tag for 6DCAA09F00007580_0000000000000001
Set tag for 1A7073F20000689C_0000000000000001
Set tag for DC4AE1650000091C_0000000000000001
Set tag for 770DCC5600008EC0_0000000000000001
Unlocking DB
Unlocked DB
Warning[extractor-c++]: In main: Extractor exiting with code 1

Maybe the logs are messed up due to concurrency. Hopefully It is useful.

[jketema]

Maybe the logs are messed up due to concurrency. Hopefully It is useful.

No, this is not useful. If there are lines like "3 errors detected in the compilation of "xxx.cpp".", that means there should also be lines that contain the text "error: ". Those are suspiciously absent.