GraalVM 22.3.0 JDK - code cache is full error

[guraaji]

code cache is full error

We have recently updated to GraalVM 22.3.0 as our JDK, and since then have seen a couple of production alerts with a stack trace. Here is the stack trace:

Caused by: org.graalvm.polyglot.PolyglotException: java.lang.RuntimeException: jdk.vm.ci.code.BailoutException: Code installation failed: code cache is full
at jdk.vm.ci.hotspot.HotSpotCodeCacheProvider.installCode(HotSpotCodeCacheProvider.java:149)
at jdk.vm.ci.code.CodeCacheProvider.setDefaultCode(CodeCacheProvider.java:67)
at org.graalvm.compiler.truffle.compiler.hotspot.HotSpotTruffleCompilerImpl.compileAndInstallStub(HotSpotTruffleCompilerImpl.java:263)
at org.graalvm.compiler.truffle.compiler.hotspot.HotSpotTruffleCompilerImpl.installTruffleReservedOopMethod(HotSpotTruffleCompilerImpl.java:237)
at org.graalvm.compiler.truffle.compiler.hotspot.libgraal.TruffleToLibGraalEntryPoints.installTruffleReservedOopMethod(TruffleToLibGraalEntryPoints.java:319)
at org.graalvm.compiler.truffle.runtime.hotspot.libgraal.TruffleToLibGraalCalls.installTruffleReservedOopMethod(Native Method)
at org.graalvm.compiler.truffle.runtime.hotspot.libgraal.LibGraalHotSpotTruffleCompiler.installTruffleReservedOopMethod(LibGraalHotSpotTruffleCompiler.java:169)
at org.graalvm.compiler.truffle.runtime.hotspot.AbstractHotSpotTruffleRuntime.installReservedOopMethods(AbstractHotSpotTruffleRuntime.java:545)
at org.graalvm.compiler.truffle.runtime.hotspot.AbstractHotSpotTruffleRuntime.bypassedReservedOop(AbstractHotSpotTruffleRuntime.java:514)
at org.graalvm.compiler.truffle.runtime.hotspot.HotSpotFastThreadLocal.setJVMCIReservedReference(HotSpotFastThreadLocal.java:139)
at org.graalvm.compiler.truffle.runtime.hotspot.HotSpotFastThreadLocal.set(HotSpotFastThreadLocal.java:62)
at com.oracle.truffle.polyglot.PolyglotFastThreadLocals.leave(PolyglotFastThreadLocals.java:139)
at com.oracle.truffle.polyglot.PolyglotThreadInfo.leaveInternal(PolyglotThreadInfo.java:158)
at com.oracle.truffle.polyglot.PolyglotEngineImpl.leaveCached(PolyglotEngineImpl.java:2080)
at com.oracle.truffle.polyglot.PolyglotEngineImpl.leave(PolyglotEngineImpl.java:2055)
at com.oracle.truffle.polyglot.PolyglotEngineImpl.leaveIfNeeded(PolyglotEngineImpl.java:1976)
at com.oracle.truffle.polyglot.PolyglotValueDispatch.hostLeave(PolyglotValueDispatch.java:1241)
at com.oracle.truffle.polyglot.PolyglotContextImpl.eval(PolyglotContextImpl.java:1484)
at com.oracle.truffle.polyglot.PolyglotContextDispatch.eval(PolyglotContextDispatch.java:63)
at org.graalvm.polyglot.Context.eval(Context.java:399)
at com.cdk.dna.rules.js.JavascriptExecutorGraal.transform(JavascriptExecutorGraal.java:52)
at com.cdk.dna.rules.js.JavascriptRuleExecutionService.executeAllActiveRules(JavascriptRuleExecutionService.java:107)
at com.cdk.dna.rules.js.JavascriptRuleExecutionService.executeAllActiveRules(JavascriptRuleExecutionService.java:84)
at com.cdk.dna.rules.js.JavascriptRuleExecutionService.executeAllActiveRules(JavascriptRuleExecutionService.java:25)
at com.cdk.dna.rules.js.JavascriptRuleServiceFactory.executeAllActiveRules(JavascriptRuleServiceFactory.java:87)
at com.cdk.dna.rules.js.JavascriptRuleServiceFactory.executeAllActiveRules(JavascriptRuleServiceFactory.java:41)
at com.cdk.dna.rules.kafka.consumers.RuleExecutionTransformer.transform(RuleExecutionTransformer.java:94)
at com.cdk.dna.rules.kafka.consumers.RecordProcessor.processInboundRuleset(RecordProcessor.java:219)
at com.cdk.dna.rules.kafka.consumers.RecordProcessor.lambda$handlePartitionRecords$8(RecordProcessor.java:145)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:960)
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:934)
at java.util.stream.AbstractTask.compute(AbstractTask.java:327)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:754)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:686)
at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:927)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at com.cdk.dna.rules.kafka.consumers.RecordProcessor.handlePartitionRecords(RecordProcessor.java:149)
at com.cdk.dna.kafka.consumer.SynchronousPartitionRecordsHandlerImpl.lambda$handle$0(SynchronousPartitionRecordsHandlerImpl.java:19)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
at util.TokenAwareRunnable.run(TokenAwareRunnable.java:28)
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

Instructions regarding code caching are here: https://www.graalvm.org/22.0/reference-manual/embed-languages/#code-caching-across-multiple-contexts

It's written that you must use:

- A shared engine (yes, we do)
- A context configured to that engine (yes, we do, one context per thread in a ThreadLocal)
- A Source object with cached set to true (we explicitly disable code caching by setting this to false, due to previous memory issues – I think because we don't re-use the Source objects)

The javadoc of "cached(boolean)" clearly states that it controls code caching, so we're certain we have the right method.

  Enables or disables code caching for this source. By default code 
  caching is enabled. If true then the source does not 
  require parsing every time this source is evaluated. 
  If false then the source requires parsing every time the 
  source is evaluated but does not remember any state. Disabling caching 
  may be useful if the source is known to only be evaluated once.


  If a source instance is no longer referenced by the client then all code 
  caches will be freed automatically. Also, if the underlying context or 
  engine is no longer referenced then cached code for evaluated sources 
  will be freed automatically.

So bottom line is:

- We appear to be disabling code cache, and we did so to fix a prior memory problem, which was fixed by flipping that flag
- We deployed 22.3.0 and now we get an exception suggesting the code cache is full

[oubidar-Abderrahim]

Hi, Thank you for reaching out about this, could you please share a reproducer example for this issue?

[timgstewart]

Hi, I'm a senior developer on the team working with Ajinkya who originally opened this defect.

I've since realized that what we are disabling is a sources cache, and the BailoutException we are seeing is related to the JVM's code cache. If I understand, these are two different things.

It looks like when truffle compiles each javascript snippet it tries to use that code cache. Our environment uses Javascript as an expression language to write processing rules, and there are thousands of variations of the scripts at runtime executed many thousands of times per minute/second.

Here is a simple app that simulates what we are doing:

`import java.util.function.BiFunction;
import java.util.stream.IntStream;
import org.graalvm.polyglot.Context;
import org.graalvm.polyglot.Engine;
import org.graalvm.polyglot.HostAccess;
import org.graalvm.polyglot.Source;
import org.graalvm.polyglot.Value;

public class GraalVmCodeCacheTest {

private final static String scriptTemplate = "function(v) { v.dealerId = \"{VALUE}\"; return v; }";

public static void  main(String[] args) throws Exception {
    final Engine engine = Engine.newBuilder().build();
    final ThreadLocal<Context> polyglotContexts = ThreadLocal.withInitial(() -> {
        Context c = Context.newBuilder("js")
                // Allow scripts to access all methods on the JsHelper class 
                // (and everything else since it returns java objects)
                .allowHostAccess(HostAccess.ALL)
                .allowHostClassLookup(className -> true)
                .engine(engine).build();
        return c;
    });;

    BiFunction<String, String, String> executor = (String val, String func) -> {
        Context context = null;
        try {
            context = polyglotContexts.get();
            Value jsBindings = context.getBindings("js");
            //jsBindings.putMember("jsHelper", JsHelperFactory.instance().createHelper(null));
            jsBindings.putMember("dataObj", val);
            Source source = Source.newBuilder("js", "JSON.stringify(" + func + "(JSON.parse(dataObj)))", "rule.js")
                    // turns out that source caching appears to use significant amounts of memory (but frees it when heap gets full)
                    // and according to tests run, actually had lower performance than not caching.
                    .cached(false)
                    .buildLiteral();
            Value result = context.eval(source);
            return result.asString();
        } catch (RuntimeException | Error ex) {
            if (context != null) {
                context.close();
                polyglotContexts.remove();
            }
            throw ex;
        } finally {
        }
    };

    System.out.println("start perf test on executor...");

    IntStream.range(0, Integer.MAX_VALUE).parallel().forEach(i -> {
        try {
            if (i%1000==0) {
                System.out.println("Begin iteration "+i);
            }
            String data = "{ \"dealerId\" : \"" + Integer.toString(i) + "\" }";
            final String func = scriptTemplate.replace("{VALUE}", Integer.toString(i-1));
            executor.apply(data, func);
        } catch (org.graalvm.polyglot.PolyglotException ex) {
            System.out.println("Bailout exception reached on iteration: " + i);
            ex.printStackTrace();
            System.exit(1);
        }
    });
}

}
`

I've been able to reach the "Bailout execption reached on iteration" line by executing this with a low code cache setting:

javac GraalVmCodeCacheTest.java && java -Xmx50M -XX:ReservedCodeCacheSize=3500K -XX:InitialCodeCacheSize=500k -cp . GraalVmCodeCacheTest

Within a few seconds I receive:

Bailout exception reached on iteration: 529008093 org.graalvm.polyglot.PolyglotException: java.lang.RuntimeException: jdk.vm.ci.code.BailoutException: Error installing Stub<ExceptionHandlerStub.exceptionHandler>: code cache is full

I hope that helps, I believe this is the same condition occuring in production, but in our production environment it runs for a much longer time. Eventually the error occurs, and our kubernetes pod restarts itself. It takes some time for the pod to start back up and recover, but we continue processing after that.

Is there any way to disable code caching for Javascript code running in the Polyglot engine?

[timgstewart]

Also in the 22.1.x release this did not happen, it only began to occur upon moving to the 22.3.0 release.

If I repeat the same test case above on the 22.1.0 VM, for example, the problem does not occur.

[GracefulAncestralCode]

I have the same problem with 22.2.0 release

[oubidar-Abderrahim]

If this issue is still relevant on Oracle GraalVM for JDK17 or for JDK21, please create a new issue or reopen this one. Thank you