Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read segment info from latest commit whenever possible #11361

Merged

Conversation

Projects
None yet
4 participants
@dakrone
Copy link
Member

commented May 26, 2015

Instead of listing the directory to file the latest segments_N file, we
should re-use the generation/filename from the last commit. This allows
us to avoid potential race conditions on the filesystem as well as
reduce the number of directory listings performed.

@dakrone

This comment has been minimized.

Copy link
Member Author

commented May 26, 2015

@mikemccand can you take a look at this?

@@ -152,7 +152,7 @@ public SegmentInfos readLastCommittedSegmentsInfo() throws IOException {
*/
private static SegmentInfos readSegmentsInfo(IndexCommit commit, Directory directory) throws IOException {
try {

This comment has been minimized.

Copy link
@mikemccand

mikemccand May 26, 2015

Contributor

Maybe remove Directory directory arg here? Or assert that directory == commit.getDirectory()?

@mikemccand

This comment has been minimized.

Copy link
Contributor

commented May 26, 2015

LGTM, thanks @dakrone!

@dakrone dakrone force-pushed the dakrone:read-si-from-searcher-commit-on-start branch from 41c9a61 to 6646881 May 26, 2015

[CORE] Read segment info from latest commit whenever possible
Instead of listing the directory to file the latest segments_N file, we
should re-use the generation/filename from the last commit. This allows
us to avoid potential race conditions on the filesystem as well as
reduce the number of directory listings performed.

@dakrone dakrone merged commit 6646881 into elastic:master May 26, 2015

1 check passed

CLA Commit author has signed the CLA
Details

@kevinkluge kevinkluge removed the review label May 26, 2015

@dakrone dakrone removed the v1.6.0 label May 27, 2015

@dakrone

This comment has been minimized.

Copy link
Member Author

commented May 27, 2015

I don't think I'm going to backport this, the Lucene API is different and I think I will work on an alternative to avoid upgrading Lucene 3.x segments when not needed, which is when the race condition usually occurs.

dakrone added a commit to dakrone/elasticsearch that referenced this pull request May 28, 2015

Fall back to reading SegmentInfos from Store if reading from commit
fails

In the event that reading from the latest commit fails, we should fall
back to reading from the `Store` using the traditional
`Directory.listAll()`

Related to elastic#11361

dakrone added a commit to dakrone/elasticsearch that referenced this pull request May 28, 2015

Fall back to reading SegmentInfos from Store if reading from commit
fails

In the event that reading from the latest commit fails, we should fall
back to reading from the `Store` using the traditional
`Directory.listAll()`

Related to elastic#11361

dakrone added a commit to dakrone/elasticsearch that referenced this pull request May 28, 2015

Fall back to reading SegmentInfos from Store if reading from commit
fails

In the event that reading from the latest commit fails, we should fall
back to reading from the `Store` using the traditional
`Directory.listAll()`

Related to elastic#11361

@dakrone dakrone deleted the dakrone:read-si-from-searcher-commit-on-start branch Jun 1, 2015

@clintongormley clintongormley changed the title [CORE] Read segment info from latest commit whenever possible Read segment info from latest commit whenever possible Jun 8, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.