LUCENE-8571: Don't block on FrozenBufferedUpdates#apply during IW#processEvents #503
Conversation
|
@mikemccand can you take a look |
|
I recommend to look at the diff with |
jpountz
left a comment
jpountz
left a comment
There was a problem hiding this comment.
I had a look out of curiosity but I'm not familiar enough with this class to comment, I'll leave it to Mike. :)
lucene/core/src/java/org/apache/lucene/index/FrozenBufferedUpdates.java
Outdated
Show resolved
Hide resolved
…cessEvents While indexing we try to apply frozen deletes packages concurrently on indexing threads if necessary. This is done in an opaque way via IndexWriter#processEvents. Yet, when we commit or refresh we have to ensure we apply all frozen update packages before we return. Today we execute the apply method in a blocking fashion which is unncessary when we are in a IndexWriter#processEvents context, we block indexing threads while they could just continue since it's already being applied. We also might wait in BufferedUpdatesStream when we apply all necessary updates were we can continue with other work instead of waiting. This change also tries to apply the packages that are not currently applied first in order to not unnecessarily block.
s1monw
force-pushed
the
optimize_apply_frozen_updates
branch
from
12a878e to
1203118
Compare
|
pushed |
|
I'm curious if you were able to measure a good improvement from this? I guess a large delete concurrent with indexing lots of docs would stress it? |
|
@msokolov I measures a pretty signficant lock contention on this lock without my change. with the change it went down significantly. Refreshes got 30% faster for that benchmark (small docs with 25% collision on the IDs and soft-deletes enabled). There was some minor indexing throughput improvements but nothing major. |
4 participants
While indexing we try to apply frozen deletes packages concurrently
on indexing threads if necessary. This is done in an opaque way via
IndexWriter#processEvents. Yet, when we commit or refresh we have to
ensure we apply all frozen update packages before we return.
Today we execute the apply method in a blocking fashion which is unnecessary
when we are in a IndexWriter#processEvents context, we block indexing
threads while they could just continue since it's already being applied.
We also might wait in BufferedUpdatesStream when we apply all necessary updates
were we can continue with other work instead of waiting.
This change also tries to apply the packages that are not currently applied
first in order to not unnecessarily block.