General issue: getting error code 100 after increasing CodeQL memory space to 16 GB RAM

[auroSymplr]

I am adding CodeQL checks to our repository for it to run when code is pushed to our branch. On doing that I got the following error.

  A fatal error occurred: Severe disk cache trouble (corruption or out of space) at /home/runner/work/_temp/codeql_databases/csharp/db-csharp/default/cache/relations/29.pack: Failed to write item to disk
  (eventual cause: IOException "No space left on device")
  Error: Failure invoking /opt/hostedtoolcache/CodeQL/0.0.0-20220908/x64/codeql/codeql with arguments database,run-queries,--ram=5920,--threads=2,/home/runner/work/_temp/codeql_databases/csharp,--min-disk-free=1024,-v,/home/runner/work/_temp/codeql_databases/csharp-queries-builtin.qls.
  
        Exit code 2 and error was:
  
        Writing logs to /home/runner/work/_temp/codeql_databases/csharp/log/database-run-queries-20221003.092700.585.log.

Upon increasing the space upto 16 GB, I am getting this error

Starting evaluation of codeql\csharp-queries\Telemetry\UnsupportedExternalAPIs.ql.
  Oops! A fatal internal error occurred.
  com.semmle.util.exception.CatastrophicError: An error occurred during the evaluation of Element#5e6df1e8::NamedElement::getQualifiedName#0#dispred#ff/2@i47#15be9x52
  The RA to evaluate was:
  
      {3} r1 = SELECT Element#5e6df1e8::NamedElement::hasQualifiedName#2#dispred#fff#prev_delta ON In.1 != ""
      {2} r2 = SCAN r1 OUTPUT In.0, (In.1 ++ "." ++ In.2)
  
      {3} r3 = SCAN Element#5e6df1e8::NamedElement::hasQualifiedName#2#dispred#fff#prev_delta OUTPUT In.1, In.0, In.2
      {2} r4 = JOIN r3 WITH const_ ON FIRST 1 OUTPUT Lhs.1, Lhs.2
  
      {2} r5 = r2 UNION r4
      {2} r6 = r5 AND NOT Element#5e6df1e8::NamedElement::getQualifiedName#0#dispred#ff#prev(Lhs.0, Lhs.1)
      return r6
  
  (eventual cause: CatastrophicError "Tried to grow a pool located at C:\actions-runner\_work\_temp\codeql_databases\c...")
  	at com.semmle.inmemory.eval.Evaluate.reportError(Evaluate.java:402)
  	at com.semmle.inmemory.eval.Evaluate.evaluate(Evaluate.java:149)
  	at com.semmle.inmemory.relations.RecursiveRelationEvaluator.evaluateSingleJob(RecursiveRelationEvaluator.java:257)
  	at com.semmle.inmemory.relations.RecursiveRelationEvaluator.evaluateIteration(RecursiveRelationEvaluator.java:197)
  	at com.semmle.inmemory.relations.RecursiveRelationEvaluator.evaluateTo(RecursiveRelationEvaluator.java:152)
  	at com.semmle.inmemory.scheduler.RecursiveLayer.evaluate(RecursiveLayer.java:96)
  	at com.semmle.inmemory.scheduler.LayerTask.evaluate(LayerTask.java:250)
  	at com.semmle.inmemory.scheduler.ComputingTask.doWork(ComputingTask.java:632)
  	at com.semmle.inmemory.scheduler.ComputingTask.doWork(ComputingTask.java:62)
  	at com.semmle.inmemory.scheduler.DemandableTask.doWork(DemandableTask.java:[378](https://github.com/symplr-software/pdm-frmmgr-cdb-service/actions/runs/3172378279/jobs/5166787048#step:5:380))
  	at com.semmle.inmemory.scheduler.ExecutionScheduler.runnerMain(ExecutionScheduler.java:568)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  	at java.base/java.lang.Thread.run(Unknown Source)
  Caused by: com.semmle.util.exception.CatastrophicError: Tried to grow a pool located at C:\actions-runner\_work\_temp\codeql_databases\csharp\db-csharp\default\strings\0\pageDump beyond the maximum number of pages allowed. This either indicates the database is too large for CodeQL to handle, or something has gone wrong in the query being evaluated.
  	at com.semmle.inmemory.util.DiskWritableStringStore.addPage(DiskPool.java:526)
  	at com.semmle.inmemory.util.DiskWritableStringStore.add(DiskPool.java:557)
  	at com.semmle.inmemory.util.SequentialDiskPool.insertBucket(SequentialDiskPool.java:40)
  	at com.semmle.inmemory.util.DiskPool.getId(DiskPool.java:254)
  	at com.semmle.inmemory.stringpool.IntensionalGeneration.encodeString(IntensionalGeneration.java:75)
  	at com.semmle.inmemory.stringpool.StringPool.encodeString(StringPool.java:173)
  	at com.semmle.inmemory.eval.ExprEvaluator$MakingString.evalToLong(ExprEvaluator.java:170)
  	at com.semmle.inmemory.eval.ExprEvaluator$BatchEvaluator.evaluateBatch(ExprEvaluator.java:261)
  	at com.semmle.inmemory.eval.RenameEvaluator.addBatch(RenameEvaluator.java:23)
  	at com.semmle.inmemory.relations.TupleBatch.startNewTuple(TupleBatch.java:169)
  	at com.semmle.inmemory.eval.BatchingSink.addTuple(BatchingSink.java:56)
  	at com.semmle.inmemory.eval.TestingSink.addBatch(TestingSink.java:56)
  	at com.semmle.inmemory.eval.SplitterSink.addBatch(SplitterSink.java:44)
  	at com.semmle.inmemory.relations.TupleBatch.startNewTuple(TupleBatch.java:169)
  	at com.semmle.inmemory.relations.BaseIntArrayRelation.read(BaseIntArrayRelation.java:37)
  	at com.semmle.inmemory.relations.BaseIntArrayRelation.batchedMap(BaseIntArrayRelation.java:110)
  	at com.semmle.inmemory.caching.PagedRelation.batchedMap(PagedRelation.java:175)
  	at com.semmle.inmemory.relations.DeduplicatingUnionRelation.batchedMap(DeduplicatingUnionRelation.java:181)
  	at com.semmle.inmemory.pipeline.AbstractRelationPipeline.runInternal(AbstractRelationPipeline.java:29)
  	at com.semmle.inmemory.pipeline.Pipeline.run(Pipeline.java:135)
  	at com.semmle.inmemory.pipeline.TupleOperationPipeline.runInternal(TupleOperationPipeline.java:24)
  	at com.semmle.inmemory.pipeline.Pipeline.run(Pipeline.java:124)
  	at com.semmle.inmemory.pipeline.JoinPipeline.runInternal(JoinPipeline.java:31)
  	at com.semmle.inmemory.pipeline.Pipeline.run(Pipeline.java:124)
  	at com.semmle.inmemory.pipeline.UnionPipeline.runInternal(UnionPipeline.java:23)
  	at com.semmle.inmemory.pipeline.Pipeline.run(Pipeline.java:124)
  	at com.semmle.inmemory.pipeline.SetOpPipeline.runInternal(SetOpPipeline.java:26)
  	at com.semmle.inmemory.pipeline.Pipeline.run(Pipeline.java:124)
  	at com.semmle.inmemory.eval.Evaluate.evaluate(Evaluate.java:141)
  	... 12 more
  Error: Failure invoking C:\actions-runner\_work\_tool\CodeQL\0.0.0-20220923\x64\codeql\codeql.exe with arguments database,run-queries,--ram=14846,--threads=4,C:\actions-runner\_work\_temp\codeql_databases\csharp,--min-disk-free=1024,-v,C:\actions-runner\_work\_temp\codeql_databases\csharp-queries-builtin.qls.
  
        Exit code 100 and error was:
  
        Writing logs to C:\actions-runner\_work\_temp\codeql_databases\csharp\log\database-run-queries-20221003.074135.084.log.

Not sure what is the issue here. Could any one help??

[hvitved]

I am guessing that the second error also means "no space left", but let someone with more knowledge about the evaluator chip in.

[hvitved]

Would it be possible to further increase disk space and rerun the analysis?

[auroSymplr]

We have increased the RAM to 16 GB. The disk space is already more than 120 GB. Just a side note: the application on which CodeQL is enabled is not a very momory intensive application. It is like a broker service which is not more than 80 MB.

Edit: we tried the same with 64 GB and 128 GB RAM as well. getting the same error.