Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore segments.gen on metadata snapshots #7379

Closed
wants to merge 2 commits into from

Conversation

s1monw
Copy link
Contributor

@s1monw s1monw commented Aug 21, 2014

The segments.gen file is optional and might even change while we read it. It's safer to just ignore that file in the snapshot instead.

we saw test failing with:

Caused by: org.apache.lucene.index.CorruptIndexException: codec footer mismatch: actual footer=0 vs expected footer=-1071082520 (resource: MockIndexInputWrapper(NIOFSIndexInput(path="/Users/igor/Projects/elasticsearch/data/shared-hoverfly.local-CHILD_VM=[0]-CLUSTER_SEED=[-3070528509192838244]-HASH=[138C74D8B6080338]/nodes/1/indices/test-idx/5/index/segments.gen")))
    at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:235)
    at org.apache.lucene.codecs.CodecUtil.retrieveChecksum(CodecUtil.java:228)
    at org.elasticsearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:564)
    at org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:498)
    at org.elasticsearch.index.store.Store$MetadataSnapshot.<init>(Store.java:459)
    at org.elasticsearch.index.store.Store.getMetadata(Store.java:154)
    at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:428)

which can happen if commits happen concurrently since this file gets corrupted on purpose during a commit

@rmuir
Copy link
Contributor

rmuir commented Aug 21, 2014

My concern is that this is just a workaround, if metadata code isn't properly reading the latest commit, then this is just going to push the problem down to segments_N logic.

How is this code looking for the latest commit? Maybe it should use SegmentInfos.read/FindSegmentsFile instead?

@s1monw
Copy link
Contributor Author

s1monw commented Aug 21, 2014

well the reason for this to fail here is that I already successfully read the .si file and then I try to read the checksum of the segments.gen file after I read the commit. but in that window it can change... so I think it's just a misunderstanding?

@rmuir
Copy link
Contributor

rmuir commented Aug 21, 2014

My bad, everything is good as of this morning. we read commit correctly, and read the correct one. So I agree, just ignore it.

The segments.gen file is optional and might even change while we
read it. It's safer to just ignore that file in the snapshot instead.
@imotov
Copy link
Contributor

imotov commented Aug 21, 2014

LGTM

@s1monw
Copy link
Contributor Author

s1monw commented Aug 21, 2014

pushed!

@s1monw s1monw closed this Aug 21, 2014
@s1monw s1monw deleted the ignore_segments_gen branch August 21, 2014 16:11
@jpountz jpountz removed the review label Aug 26, 2014
@clintongormley clintongormley changed the title [STORE] Ignore segments.gen on metadata snapshots Internal: Ignore segments.gen on metadata snapshots Sep 8, 2014
@clintongormley clintongormley added the :Distributed/Store Issues around managing unopened Lucene indices. If it touches Store.java, this is a likely label. label Jun 7, 2015
@clintongormley clintongormley changed the title Internal: Ignore segments.gen on metadata snapshots Ignore segments.gen on metadata snapshots Jun 7, 2015
@clintongormley clintongormley added :Distributed/Engine Anything around managing Lucene and the Translog in an open shard. and removed :Distributed/Store Issues around managing unopened Lucene indices. If it touches Store.java, this is a likely label. labels Feb 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug :Distributed/Engine Anything around managing Lucene and the Translog in an open shard. v1.3.3 v1.4.0.Beta1 v2.0.0-beta1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants