-
Notifications
You must be signed in to change notification settings - Fork 2.6k
LUCENE-9406: Add IndexWriterEventListener to track events in IndexWriter #2342
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LUCENE-9406: Add IndexWriterEventListener to track events in IndexWriter #2342
Conversation
MergePolicy.OneMerge nextMerge = null; | ||
|
||
if (pendingMerges.size() > 0) { | ||
// nocommit getting OneMerge instance here via mergeSource.getNextMerge() will |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about passing the pendingMerges
to the event listener instead? (Sorry if I asked for OneMerge
on the issue! MergeSpecification
is better since it can hold multiple merges, allows event listener to log things like how many merges were requested during commit
, etc.).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No problem! From the context I assume you are actually meaning to pass pointInTimeMerges
into event listener, since pendingMerges
is of type Deque<MergePolicy.OneMerge>
? I've pushed a commit to use pointInTimeMerges
for now.
} | ||
|
||
// Test basic semantics of merge on commit and events recording invocation | ||
public void testMergeOnCommitWithEventListener() throws IOException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You might also edit LuceneTestCase.newIndexWriterConfig
to randomly swap in a MockIndexWriterEventListener
just to exercise this listener in any tests using that API, which is quite a few. It could uncover times when we accidentally break something when this listener is invoked ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense. Added.
…r randomly into LuceneTestCase.newIndexConfig
I think it's good overall but I'm wondering whether it makes sense to make that field volatile... do we want to allow changing listeners over index writer lifecycle? I think it should be a regular field and IW should just read it once (and set forever). |
Thanks for the feedback Dawid! I thought about this a bit as well when I noticed most of the other fields are volatile, but I can't decide for sure which direction to go as I don't have enough context information about the different use cases IW may need to support in the wild, so I ended up following the existing pattern here. However, I do feel that it may not hurt for application to have the freedom to switch event listener in the middle if situation or need dictates (i.e. a service application containing Lucene may need to switch event listener if it exposes an API for its client to choose how the IW event stream to be sent and stored) ? |
If such a situation arises folks will have the ability to write a custom proxy listener with the ability to switch, add and/or remove delegates. This keeps the freedom to do anything you said and at the same time simplifies implementation (and concurrency-issues) on IW side? |
Ah that's very true! I've pushed a commit to remove volatile then. Please let me know if it looks good. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, looks good to me!
Circling back to this PR as I just realized I have forgotten about it.. :D Thanks @dweiss for the review! @mikemccand do you think this PR is ready now? Anything else you would like me to improve on? |
Looks good to me. I've merged this in, sorry for the delay. |
No worry! Thanks for the review and merge! |
Woops, sorry for the belated response, and thank you @zacharymorn for creating this and @dweiss for merging -- it looks great! We can now add other events to track incrementally over time ... |
No problem, and thanks for the review feedback as well Michael! |
Description / Solution
Add IndexWriterEventListener interface to track events in IndexWriter
Tests
Note: precommit is currently failing due to nocommit comment
Checklist
Please review the following and check all that apply:
master
branch../gradlew check
.