Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix the race condition in realtime text index refresh thread #6858

Closed
GSharayu opened this issue Apr 28, 2021 · 1 comment
Closed

Fix the race condition in realtime text index refresh thread #6858

GSharayu opened this issue Apr 28, 2021 · 1 comment

Comments

@GSharayu
Copy link
Contributor

The refresh thread maintains a single global queue of realtime text index readers across all consuming segments across all tables.

A reader is polled, refreshed and supposed to be added back to the queue so that it can be refreshed again in the next cycle.
There is a race condition: If we find the queue empty, but something gets added before we lock the mutex, the thread signal will be lost and it will be in await mode until next segment signals.

Set of steps for race condition

  1. Thread finds queue empty and enters the while check on https://github.com/apache/incubator-pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneIndexReaderRefreshThread.java#L69
  2. Meanwhile another thread grabs lock at https://github.com/apache/incubator-pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneIndexRefreshState.java#L86
  3. The current thread adds the segment in queue and signals the condition variable (right now its not hold)
  4. The lock after released from step 3 will be grabbed by thread in step 1
  5. It will then await for signal on condition variable(Which is bug!) and might have to wait until some other segment signals it
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants