General issue: Randomly happens com.semmle.util.exception.CatastrophicError: End of file found while looking for '.pop' with a C++ project #11586

mrjohannchang · 2022-12-07T06:02:01Z

The symptom is pretty much the same with #9929 but the issue does not happen 100% of the time.

CodeQL CLI version: 2.6.3
CodeQL queries version: lgtm/1.29.0
Windows Driver Developer Supplemental Tools version: WHCP_22H2

Finalizing database at C:\Users\user\workspace\ipu-cam-drv-ice-master\camerasw\bins\Release.x64\CodeQL\Database.
ipu_device_ff_properties_impl.h.4f534201.trap (pg_187_control_init.c.86ad7c77_0.trap.tar.br) for 2 link targets, 1605: java.io.IOException: Brotli stream decoding failed
org.brotli.dec.BrotliInputStream.read(BrotliInputStream.java:151)
org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:608)
java.base/java.io.FilterInputStream.read(Unknown Source)
com.semmle.inmemory.trap.TrapInputStream.read(TrapInputStream.java:61)
com.semmle.inmemory.trap.TrapScanner.fill(TrapScanner.java:507)
com.semmle.inmemory.trap.TrapScanner.ensureNext(TrapScanner.java:484)
com.semmle.inmemory.trap.TrapScanner.identOrLabelRest(TrapScanner.java:171)
com.semmle.inmemory.trap.TrapScanner.nextToken(TrapScanner.java:123)
com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:491)
com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:423)
com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:191)
com.semmle.inmemory.trap.ImportTasksProcessor.lambda$null$1(ImportTasksProcessor.java:151)
com.semmle.util.concurrent.FutureUtils.lambda$null$8(FutureUtils.java:136)
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base/java.lang.Thread.run(Unknown Source)

 ... caused by:

org.brotli.dec.BrotliRuntimeException: Invalid backward reference
org.brotli.dec.Decode.decompress(Decode.java:887)
org.brotli.dec.BrotliInputStream.read(BrotliInputStream.java:145)
org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:608)
java.base/java.io.FilterInputStream.read(Unknown Source)
com.semmle.inmemory.trap.TrapInputStream.read(TrapInputStream.java:61)
com.semmle.inmemory.trap.TrapScanner.fill(TrapScanner.java:507)
com.semmle.inmemory.trap.TrapScanner.ensureNext(TrapScanner.java:484)
com.semmle.inmemory.trap.TrapScanner.identOrLabelRest(TrapScanner.java:171)
com.semmle.inmemory.trap.TrapScanner.nextToken(TrapScanner.java:123)
com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:491)
com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:423)
com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:191)
com.semmle.inmemory.trap.ImportTasksProcessor.lambda$null$1(ImportTasksProcessor.java:151)
com.semmle.util.concurrent.FutureUtils.lambda$null$8(FutureUtils.java:136)
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base/java.lang.Thread.run(Unknown Source)
program_psa_out_s2v_dma_control_init.c.e495e4e7.trap (program_psa_out_s2v_dma_control_init.c.e495e4e7_0.trap.tar.br) for 2 link targets, 3669: java.io.IOException: Brotli stream decoding failed
org.brotli.dec.BrotliInputStream.read(BrotliInputStream.java:151)
org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:608)
java.base/java.io.FilterInputStream.read(Unknown Source)
com.semmle.inmemory.trap.TrapInputStream.read(TrapInputStream.java:61)
com.semmle.inmemory.trap.TrapScanner.fill(TrapScanner.java:507)
com.semmle.inmemory.trap.TrapScanner.ensureNext(TrapScanner.java:484)
com.semmle.inmemory.trap.TrapScanner.stringBody(TrapScanner.java:191)
com.semmle.inmemory.trap.TrapScanner.string(TrapScanner.java:224)
com.semmle.inmemory.trap.TrapScanner.nextToken(TrapScanner.java:119)
com.semmle.inmemory.trap.TRAPReader.scanLabelValue(TRAPReader.java:790)
com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:503)
com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:423)
com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:191)
com.semmle.inmemory.trap.ImportTasksProcessor.lambda$null$1(ImportTasksProcessor.java:151)
com.semmle.util.concurrent.FutureUtils.lambda$null$8(FutureUtils.java:136)
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base/java.lang.Thread.run(Unknown Source)

 ... caused by:

org.brotli.dec.BrotliRuntimeException: Corrupted padding bits
org.brotli.dec.BitReader.jumpToByteBoundary(BitReader.java:178)
org.brotli.dec.Decode.decompress(Decode.java:1004)
org.brotli.dec.BrotliInputStream.read(BrotliInputStream.java:145)
org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:608)
java.base/java.io.FilterInputStream.read(Unknown Source)
com.semmle.inmemory.trap.TrapInputStream.read(TrapInputStream.java:61)
com.semmle.inmemory.trap.TrapScanner.fill(TrapScanner.java:507)
com.semmle.inmemory.trap.TrapScanner.ensureNext(TrapScanner.java:484)
com.semmle.inmemory.trap.TrapScanner.stringBody(TrapScanner.java:191)
com.semmle.inmemory.trap.TrapScanner.string(TrapScanner.java:224)
com.semmle.inmemory.trap.TrapScanner.nextToken(TrapScanner.java:119)
com.semmle.inmemory.trap.TRAPReader.scanLabelValue(TRAPReader.java:790)
com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:503)
com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:423)
com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:191)
com.semmle.inmemory.trap.ImportTasksProcessor.lambda$null$1(ImportTasksProcessor.java:151)
com.semmle.util.concurrent.FutureUtils.lambda$null$8(FutureUtils.java:136)
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base/java.lang.Thread.run(Unknown Source)
Oops! A fatal internal error occurred.
com.semmle.util.exception.CatastrophicError: End of file found while looking for '.pop'.
	at com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:435)
	at com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:191)
	at com.semmle.inmemory.trap.ImportTasksProcessor.lambda$null$1(ImportTasksProcessor.java:151)
	at com.semmle.util.concurrent.FutureUtils.lambda$null$8(FutureUtils.java:136)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

microsoft/Windows-Driver-Developer-Supplemental-Tools#38

The text was updated successfully, but these errors were encountered:

hvitved · 2022-12-07T07:50:27Z

@github/codeql-c-extractor : Would you mind taking a look, please?

jketema · 2022-12-07T07:58:06Z

Hi @changyuheng

Thanks for the report. This is different from #9929. One of the files produced during tracing seems to be incomplete, causing your problem. Version 2.6.3 of the CodeQL CLI is frozen in time, so we cannot get this fixed for that version. However, I would like to know if this problem still occurs with the latest version of the CLI, so we can fix the problem in the next version if it still occurs. This won't work in combination with the supplemental tools, unfortunately.

Would you be willing to create a database with the latest CLI version, which can be found here? Then, if the failure still occurs, would you be willing to share the build-tracer.log file that is produced during database creation? That should help to diagnose this further if the problem still occurs.

Thanks.

mrjohannchang · 2022-12-07T09:53:07Z

Hi @jketema, I'm willing to create the database for you. But if the latest CodeQL CLI is not compatible with https://github.com/microsoft/Windows-Driver-Developer-Supplemental-Tools, I'm afraid I have no methods to do it.

jketema · 2022-12-07T10:07:21Z

The steps from https://learn.microsoft.com/en-us/windows-hardware/drivers/devtest/static-tools-and-codeql#building-your-codeql-database also apply to the latest CodeQL CLI. It's just the analysis step that is not likely to work, but that's not relevant here.

mrjohannchang · 2022-12-07T15:12:26Z

I see what you're saying. I guess the database will be compatible. Do you mean I can let the CI use the latest CodeQL CLI to create the database, then use the compatible CodeQL CLI to do the analysis, and whenever I see the creation of the database fails, I sent the database to you? I think I can do that. There is just another issue that complicates the process: codeql database create does not exit with an error code when it fails. The issue happens with CodeQL CLI 2.6.3; not sure if it still exists in the latest one.

lauragonzalezzz · 2022-12-07T22:40:48Z

@changyuheng It seems the failure is occurring when the database is created. To determine whether this failure persists in the newer version of CodeQL CLI, @jketema is recommending that you do the following:

Upgrade your CodeQL CLI to v2.11.5
Create a new database by running the codeql database create script
Post the build-tracer.log in a new comment here for review

If the problem does persist in v2.11.5, a bug will need to be created in CodeQL CLI to track and resolve this issue to get you unblocked.
If the problem does not persist, then the bug is resolved in the newer CodeQL CLI version, and you and I can work together to determine how to unblock you on the Windows Driver Developer Supplemental Tools side.

mrjohannchang · 2022-12-07T22:44:25Z

Thank you for the clearance. However, the issue does not happen every time, so instead of manually reproducing it, I guess I have to put it into our CI.

jketema · 2022-12-07T22:51:24Z

@changyuheng Apologies for not responding earlier. It's as @lauragonzalezzz says.

Thank you for the clearance. However, the issue does not happen every time, so instead of manually reproducing it, I guess I have to put it into our CI.

Or write a script that creates the database, checks whether the error occurred, and repeats if it doesn't. You might first want to try that with the 2.6.3 CLI to see whether you can make that work.

mrjohannchang · 2022-12-07T22:57:45Z

Thank you for replying. I can write a dedicate verification script. As I mentioned earlier it's a bit complicated. codeql database create does not return an error code when failing. So I need to either grepping the output or do the further process to check if the database was created successfully. Anyway, I think I have enough info now, I'll do the verification and let you know the result. Thank you again.

mrjohannchang · 2022-12-09T08:34:28Z

Hi @jketema, @lauragonzalezzz, and @hvitved, thank you all for your kind response.

With
CodeQL CLI version: 2.6.3
CodeQL queries version: lgtm/1.29.0
Windows Driver Developer Supplemental Tools version: WHCP_22H2
I created a loop to build my project as well as monitor the resource continuously. However, I'm not able to replicate the issue. Windows crashed once while I execute the script concurrently. I thought it was an out-of-memory issue but as I said I'm not able to replicate it.

I'll keep doing my test for a week. I'll let you know if I reproduce the issue. But I currently consider this a hardware issue, so I'm gonna close this issue for now.

Thank you so much again for your support.

jketema · 2022-12-09T08:38:29Z

Hi @changyuheng

If that didn't even work, would it be possible for you to share a copy of the build-tracer.log from a run where the error did occur? Note that the log contains references to file names, which you might not want to share.

mrjohannchang · 2022-12-09T08:58:52Z

Hi @jketema, is this the log that CodeQL CLI will generate? Unfortunately, I didn't keep the original CodeQL CLI environment. Or do you mean the log file of that problematic round I created in my build process?

jketema · 2022-12-09T09:29:47Z

Or do you mean the log file of that problematic round I created in my build process?

It's one of the files that you can find in the database directory once you've tried to create a database.

mrjohannchang · 2022-12-09T09:36:28Z

I didn't keep the database directory; I'm sorry! I consider that as part of the build results and my current config will remove results every time starting another build.

jketema · 2022-12-09T10:06:57Z

I didn't keep the database directory; I'm sorry!

No worries.

Closing this again for now, as there's not much we can do at this point. Feel free to reopen at any time.

mrjohannchang · 2022-12-10T06:28:40Z

Hi @jketema, I just replicated a similar issue on another computer. Could you please let me know what information you may like to check?

Finalizing database at C:\Users\henrych1\workspace\ipu-cam-drv\camerasw\bins\Release.x64\CodeQL\Database.
ipu_sequencer_info_descriptor.c.4cc9ce9d.trap (ipu_sequencer_info_descriptor.c.4cc9ce9d_0.trap.tar.br) for 2 link targets, 2486: com.semmle.util.exception.CatastrophicError: Unexpected token IDENTIFIERvalue:ents
com.semmle.inmemory.trap.TRAPReader.scanOneField(TRAPReader.java:920)
com.semmle.inmemory.trap.TRAPReader.scanTuple(TRAPReader.java:590)
com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:512)
com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:423)
com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:191)
com.semmle.inmemory.trap.ImportTasksProcessor.lambda$null$1(ImportTasksProcessor.java:151)
com.semmle.util.concurrent.FutureUtils.lambda$null$8(FutureUtils.java:136)
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base/java.lang.Thread.run(Unknown Source)
Oops! A fatal internal error occurred.
com.semmle.util.exception.CatastrophicError: End of file found while looking for '.pop'.
        at com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:435)
        at com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:191)
        at com.semmle.inmemory.trap.ImportTasksProcessor.lambda$null$1(ImportTasksProcessor.java:151)
        at com.semmle.util.concurrent.FutureUtils.lambda$null$8(FutureUtils.java:136)
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

jketema · 2022-12-10T07:08:30Z

If possible, I'd like a copy of the build-tracer.log file which is located in a subdirectory of the database directory, so C:\Users\henrych1\workspace\ipu-cam-drv\camerasw\bins\Release.x64\CodeQL\Database in the above case.

Note The file contains mentions of file names and possibly some references to source code we could not parse, so before sharing, please make sure these are not sensitive.

mrjohannchang · 2022-12-10T17:43:07Z

Hi @jketema, the log files build-tracer and database-create contain the file names as well as function names. Because this is a proprietary repository, I cannot share the logs with you. Sorry about that. But thanks for your support anyhow!

jketema · 2022-12-10T17:49:42Z

No worries, I was sort of expecting that. You could have a look to see if there are any catastrophic errors mentioned in the log, or whether you can find any stack traces in the build-tracer.log. That might provide a clue of what is happening.

mrjohannchang · 2022-12-10T18:52:43Z

The only catastrophic error I see in build-tracer.log is:

[E 18:13:38 5072] Starting compilation TRAP C:\Users\henrych1\workspace\ipu-cam-drv-ice-master-dev\.build\CodeQL\Database\trap\cpp/compilations/54/66490215_0.trap.br
reg_dump\src\reg_dump_generic_bridge.c:
Command-line error: invalid macro definition: ROOT_DIR_PSYS=C:\Users\henrych1\workspace\ipu-cam-drv-ice-master-dev\camerasw\Source\Camera\Platform\MTL\\PSys

1 catastrophic error detected in the compilation of "reg_dump\src\reg_dump_generic_bridge.c".
Compilation terminated.

But successful builds have the same catastrophic error in this log.

mrjohannchang · 2023-02-14T02:01:09Z

By limiting the threads I was able to run the build without any problems on a computer with 32 GB RAM installed. I also have another machine that has 128 GB RAM installed that has been running the same code building with the number of threads bound to the CPU count for more than a year without any issues. I believe the root case was the shortage of RAM.

CC: @mandyhs

mrjohannchang added the question Further information is requested label Dec 7, 2022

mrjohannchang mentioned this issue Dec 7, 2022

Randomly happens com.semmle.util.exception.CatastrophicError: End of file found while looking for '.pop' with a C++ project microsoft/Windows-Driver-Developer-Supplemental-Tools#38

Closed

jketema added awaiting-response The CodeQL team is awaiting further input or clarification from the original reporter of this issue. C++ labels Dec 7, 2022

jketema removed the awaiting-response The CodeQL team is awaiting further input or clarification from the original reporter of this issue. label Dec 8, 2022

mrjohannchang closed this as completed Dec 9, 2022

jketema reopened this Dec 9, 2022

jketema closed this as completed Dec 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

General issue: Randomly happens com.semmle.util.exception.CatastrophicError: End of file found while looking for '.pop' with a C++ project #11586

General issue: Randomly happens com.semmle.util.exception.CatastrophicError: End of file found while looking for '.pop' with a C++ project #11586

mrjohannchang commented Dec 7, 2022 •

edited

Loading

hvitved commented Dec 7, 2022

jketema commented Dec 7, 2022 •

edited

Loading

mrjohannchang commented Dec 7, 2022

jketema commented Dec 7, 2022

mrjohannchang commented Dec 7, 2022 •

edited

Loading

lauragonzalezzz commented Dec 7, 2022

mrjohannchang commented Dec 7, 2022

jketema commented Dec 7, 2022

mrjohannchang commented Dec 7, 2022

mrjohannchang commented Dec 9, 2022

jketema commented Dec 9, 2022

mrjohannchang commented Dec 9, 2022 •

edited

Loading

jketema commented Dec 9, 2022

mrjohannchang commented Dec 9, 2022

jketema commented Dec 9, 2022

mrjohannchang commented Dec 10, 2022 •

edited

Loading

jketema commented Dec 10, 2022

mrjohannchang commented Dec 10, 2022

jketema commented Dec 10, 2022

mrjohannchang commented Dec 10, 2022 •

edited

Loading

mrjohannchang commented Feb 14, 2023 •

edited

Loading

General issue: Randomly happens com.semmle.util.exception.CatastrophicError: End of file found while looking for '.pop' with a C++ project #11586

General issue: Randomly happens com.semmle.util.exception.CatastrophicError: End of file found while looking for '.pop' with a C++ project #11586

Comments

mrjohannchang commented Dec 7, 2022 • edited Loading

hvitved commented Dec 7, 2022

jketema commented Dec 7, 2022 • edited Loading

mrjohannchang commented Dec 7, 2022

jketema commented Dec 7, 2022

mrjohannchang commented Dec 7, 2022 • edited Loading

lauragonzalezzz commented Dec 7, 2022

mrjohannchang commented Dec 7, 2022

jketema commented Dec 7, 2022

mrjohannchang commented Dec 7, 2022

mrjohannchang commented Dec 9, 2022

jketema commented Dec 9, 2022

mrjohannchang commented Dec 9, 2022 • edited Loading

jketema commented Dec 9, 2022

mrjohannchang commented Dec 9, 2022

jketema commented Dec 9, 2022

mrjohannchang commented Dec 10, 2022 • edited Loading

jketema commented Dec 10, 2022

mrjohannchang commented Dec 10, 2022

jketema commented Dec 10, 2022

mrjohannchang commented Dec 10, 2022 • edited Loading

mrjohannchang commented Feb 14, 2023 • edited Loading

mrjohannchang commented Dec 7, 2022 •

edited

Loading

jketema commented Dec 7, 2022 •

edited

Loading

mrjohannchang commented Dec 7, 2022 •

edited

Loading

mrjohannchang commented Dec 9, 2022 •

edited

Loading

mrjohannchang commented Dec 10, 2022 •

edited

Loading

mrjohannchang commented Dec 10, 2022 •

edited

Loading

mrjohannchang commented Feb 14, 2023 •

edited

Loading