Use ArrayDeque for JapaneseCompletionFilter output buffering

[luizmlima]

Description

JapaneseCompletionFilter.CompletionTokenGenerator uses outputs as a FIFO queue.

The current implementation uses an ArrayList with remove(0), which shifts the remaining elements after each removal. This change replaces it with an ArrayDeque and uses addLast() and removeFirst().

There is no change to the public API or to the generated tokens, their order, offsets, or position increments.

Benchmark

I used JMH to compare the current ArrayList implementation with the proposed ArrayDeque implementation.

The benchmark runs the full JapaneseCompletionAnalyzer pipeline. For each input, it creates a token stream and consumes all generated tokens.

Both versions were tested with the same benchmark code, JDK, JVM options, hardware, and JMH configuration.

Configuration

• Mode: Average time
• Unit: microseconds per operation
• Warmup: 5 iterations of 1 second
• Measurement: 10 iterations of 1 second
• Forks: 3
• JVM: -Xms1g -Xmx1g -XX:+AlwaysPreTouch

Lower values are better.

Environment

• JDK: OpenJDK Temurin 25.0.3 LTS
• Runtime: Temurin-25.0.3+9
• OS: Pop!_OS / Linux
• Kernel: 7.0.11-76070011-generic
• Architecture: x86_64
• CPU: Intel Core i5-10300H @ 2.50 GHz
• 4 physical cores
• 8 logical CPUs
• 2 threads per core

Inputs

• 東京
• ドラえもん
• シンシンシン
• シンシンシンシンシン

The first two are short, more typical inputs. The last two were included to generate larger output batches and make the queue behavior more visible.

Results

Input	ArrayList	ArrayDeque	Difference
東京	1.309 ± 0.024 us/op	1.322 ± 0.026 us/op	+0.98%
ドラえもん	2.557 ± 0.013 us/op	2.600 ± 0.051 us/op	+1.70%
シンシンシン	4.809 ± 0.074 us/op	4.656 ± 0.075 us/op	-3.19%
シンシンシンシンシン	8.288 ± 0.138 us/op	7.743 ± 0.079 us/op	-6.57%

Negative values mean that ArrayDeque completed the operation in less time.

For the shorter inputs, the error ranges overlap, so there was no meaningful performance difference.

For the larger output batches, ArrayDeque performed better:

• about 3.2% faster for シンシンシン
• about 6.6% faster for シンシンシンシンシン

The largest case went from 8.288 ± 0.138 us/op with ArrayList to 7.743 ± 0.079 us/op with ArrayDeque.

This matches the way the collection is used. ArrayList.remove(0) shifts the remaining elements, while ArrayDeque.removeFirst() does not.

The change does not improve typical short inputs in a measurable way, but it better matches the FIFO access pattern and avoids the increasing cost of repeated removals from the beginning of an ArrayList.

Validation

The following commands completed successfully:

• ./gradlew tidy
• ./gradlew -p lucene/analysis/kuromoji test --tests "org.apache.lucene.analysis.ja.TestJapaneseCompletionFilter"
• ./gradlew -p lucene/analysis/kuromoji test --tests "org.apache.lucene.analysis.ja.TestJapaneseCompletionAnalyzer"
• ./gradlew -p lucene/analysis/kuromoji check
• ./gradlew check