JDK 21, lusearch, and Lucene "regression"

[shipilev]

Wanted to report a "problem" with current lusearch benchmark in 23.11-chopin, which so far looks like the issue in Lucene support for JDK preview features. In short, running with different JDKs shows that lusearch is substantially slower with JDK 21.

Running self-built JDKs on Graviton 3 instance with:

% shipilev-jdk/build/linux-aarch64-server-release/images/jdk/bin/java -jar dacapo-23.11-chopin.jar lusearch -n 20

...yields these results:

JDK 17-dev: 1584 msec
JDK 21-dev: 2446 msec (!!!)
JDK 22+0:   2455 msec (!!!)
JDK 22+1:   1527 msec
JDK 23-dev: 1519 msec

Bisection shows that the "regression" starts in JDK 19 with: 8282191: Implementation of Foreign Function & Memory API (Preview). We cannot run JDK 19 prior that changeset: Lucene fails with NCDFE trying to access MemorySegment. We suspect that Lucene is opting in some FFM preview features, which are slower than what it used before. Mainline JDK is fast again, but only after JDK starts to identify itself as JDK 22: openjdk/jdk@5a706fb

I suspect that Lucene that ships with 23.11-chopin actually wants JDK 22 to perform well, and the intermediate version that opts-in to JDK 19 preview features is actually slower on this test.

@ChrisHegarty, is there something we can do here? Maybe there is a feature flag that switches what Lucene is using? Maybe Dacapo should update the Lucene to some other version?

[ChrisHegarty]

What version of Lucene are you using? It seems that the MemorySegment backed mmap directory is being used by default. You can disable it by setting the following system property:

-Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false.

I'm surprised that using the MemorySegment backed mmap directory causes a perf regression. This is not what we see in our tests, so it's likely that there's something else going on here too.

[steveblackburn]

We're using 9.7.0 in the Chopin release.

[caizixian]

Yes, I can confirm that we can see the same slowdown in our setup. 2845ms in Temurin 11 (0.74% 95CI), 2825ms in Temurin 17 (0.74% 95CI), and 3238ms (1% 95CI)

[caizixian]

Setting the -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false flag indeed improves the performance on Temurin 21. It's now on par with 17 and 11.

[steveblackburn]

I just want to confirm that this problem persists with Lucene 9.10.

@ChrisHegarty is this what you expect?

I'm preparing for an upcoming release, but given what I saw with 9.10, I am now wondering whether I should be turning the flag off by default. My (strong) preference is to have Lucene run "as-is". But I'd also rather not ship the suite with a known major performance regression (I regret that I did not see this when I made the recent release).

@shipilev do you have thoughts on this?

[shipilev]

@shipilev do you have thoughts on this?

It sounds to me that there is a problem with Lucene and JDK 21, which Dacapo correctly caught. (Pats the benchmark on its back.) And JDK 21 is de-facto LTS release, which would be around for years to come. So I think this problem should be fixed in Lucene, and then Dacapo should update with new version of Lucene. If there is no good version for Lucene that solves the issue with JDK 21, Dacapo should not be forced to do anything to work around performance bugs.

[ChrisHegarty]

@ChrisHegarty is this what you expect?

We run Lucene 9.10 with JDK 21 in many scenarios WITHOUT issue. Whatever it is that you're encountering, it is not an obvious or known issue with Lucene 9.10 and JDK 21. Can you please try to determine where the performance difference is, and why enableMemorySegments=false would have any affect on it.

[steveblackburn]

I can confirm that the issue appears to have been introduced in 9.7. It is not evident in 9.6 or earlier.

I'll look into this further tomorrow.

[steveblackburn]

This seems to be the culprit: apache/lucene#12294

The next question is why we're seeing the issue and you're not.

[steveblackburn]

@ChrisHegarty

A quick look at a perf profile shows the following:

9.6

  8.30%  Query0           [JIT] tid 786719      [.] int org.apache.lucene.util.compress.LZ4.decompress(org.apache.lucene.store.DataInput, int, byte[], int)
   7.86%  Query0           [JIT] tid 786719      [.] java.nio.ByteBuffer java.nio.ByteBuffer.getArray(int, byte[], int, int)
   6.88%  Query0           [JIT] tid 786719      [.] StubRoutines (final stubs)
   4.41%  Query0           [JIT] tid 786719      [.] short org.apache.lucene.store.ByteBufferIndexInput.readShort()

9.7

  10.20%  Query0           [JIT] tid 792317      [.] int java.lang.invoke.VarHandleGuards.guard_LJ_I(java.lang.invoke.VarHandle, java.lang.Object, long, java.lang.invoke.VarHandle$AccessDescriptor)
   9.71%  Query0           [JIT] tid 792317      [.] StubRoutines (final stubs)
   8.50%  Query0           [JIT] tid 792317      [.] void org.apache.lucene.store.MemorySegmentIndexInput.readBytes(byte[], int, int)
   6.63%  Query0           [JIT] tid 792317      [.] int java.lang.invoke.MethodHandle.linkToStatic(java.lang.Object, java.lang.Object, long, java.lang.invoke.MemberName)
   5.86%  Query0           [JIT] tid 792317      [.] short java.lang.invoke.VarHandleSegmentAsShorts.get(java.lang.invoke.VarHandle, java.lang.Object, long)
   5.74%  Query0           [JIT] tid 792317      [.] int org.apache.lucene.util.compress.LZ4.decompress(org.apache.lucene.store.DataInput, int, byte[], int)
   5.23%  Query0           [JIT] tid 792317      [.] short org.apache.lucene.store.MemorySegmentIndexInput.readShort()

So the link to MemorySegments is clear.

This is with Corretto-21.0.0.35.1 (build 21+35-LTS)

I'm running DaCapo lusearch with arguments -n 20 -t 1 (20 invocations of the workload, single threaded). The relevant code is here.

This runs 10 M simple queries in a single worker thread ("Query0").

[caizixian]

Running with -Xint and profiling using VisualVM shows the full stack with and without enableMemorySegments.
This is not apparent from the above perf output since most of the methods are inline into LZ4.decompress.

So I think we learnt the following:

1. Bulk of DaCapo's query threads' time is spent on LZ4.decompress, which heavily exercise IndexInput's various read* methods.
2. The MemorySegment implementation of IndexInput seems to be inefficient on older JDKs, especially with more mutator threads (running DaCapo with -t 1 or -t 32 shows very different result with and without enableMemorySegments)

[caizixian]

I wonder whether this fixes the issue on newer JDKs openjdk/jdk@1594653

[steveblackburn]

Thanks @caizixian

@ChrisHegarty this seems fairly clear now.

In short, enabling memory segments does not play nicely with JDK 21 (as @shipilev said), and we now have a profile that shows why this is so.

As @caizixian says, from all the testing we did, you will see this if you have an intense workload with a fair amount of parallelism (I saw 2X slowdown with 24 threads, but none when running single-threaded).

I am not sure why this did not show up in your testing.

Given what we know about the pathologies of JDK 21 and the data here, I wonder if it would be better for MemorySegmentIndexInput to be off by default for JDK 21. If so, my plan for DaCapo would be to include that change in the upcoming release. If not, perhaps my best option would be to downgrade to 9.6 for the upcoming release.