Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add BWC layer to .si / segments_N hashing to identify segments accurately #7436

Merged
merged 1 commit into from Aug 26, 2014

Conversation

Projects
None yet
3 participants
@s1monw
Copy link
Contributor

s1monw commented Aug 25, 2014

Due to additional safety added in #7351 we compute now a strong hash for
.si and segments_N files which are compared during snapshot / restore.
Old snapshots don't have this hash which can cause unnecessary copying
of large amount of data. This commit adds the ability to fetch this
hash from the blob store if needed.

Closes #7434

@s1monw s1monw added the review label Aug 25, 2014

@s1monw

This comment has been minimized.

Copy link
Contributor Author

s1monw commented Aug 25, 2014

@imotov can you take a look?

@imotov

View changes

src/main/java/org/elasticsearch/index/snapshots/blobstore/BlobStoreIndexShardRepository.java Outdated
@@ -47,6 +48,7 @@
import org.elasticsearch.indices.recovery.RecoveryState;
import org.elasticsearch.repositories.RepositoryName;

import java.io.ByteArrayInputStream;

This comment has been minimized.

Copy link
@imotov

imotov Aug 26, 2014

Member

Not needed

@imotov

View changes

src/main/java/org/elasticsearch/index/snapshots/blobstore/BlobStoreIndexShardRepository.java Outdated
if (fileInfo == null || !fileInfo.isSame(md) || !snapshotFileExistsInBlobs(fileInfo, blobs)) {
logger.error("### needs snapshot: " + fileInfo + " " + (fileInfo == null ? null : fileInfo.metadata()) + " md: " + md);

This comment has been minimized.

Copy link
@imotov

imotov Aug 26, 2014

Member

Debug leftovers?

@imotov

View changes

src/main/java/org/elasticsearch/index/store/Store.java Outdated
* Computes a strong hash value for small files. Note that this method should only be used for files < 1MB
*/
public static void hashFile(BytesRef fileHash, BytesRef source) throws IOException {
final int len = (int)Math.min(1024 * 1024, source.length); // for safety we limit this to 1MB

This comment has been minimized.

Copy link
@imotov

imotov Aug 26, 2014

Member

The (int) conversion is not needed here.

@imotov

View changes

src/main/java/org/elasticsearch/index/store/Store.java Outdated
fileHash.offset = 0;
fileHash.grow(fileHash.length);
fileHash.length = len;
fileHash.copyBytes(source);

This comment has been minimized.

Copy link
@imotov

imotov Aug 26, 2014

Member

The copyBytes method is going to copy the entire source regardless of the safety limit that was set above.

@imotov

This comment has been minimized.

Copy link
Member

imotov commented Aug 26, 2014

Left a few minor comments. Otherwise LGTM.

@s1monw

This comment has been minimized.

Copy link
Contributor Author

s1monw commented Aug 26, 2014

@imotov updated the PR

@imotov

This comment has been minimized.

Copy link
Member

imotov commented Aug 26, 2014

LGTM

@s1monw s1monw force-pushed the s1monw:issues/7434 branch Aug 26, 2014

[SNAPSHOT] Add BWC layer to .si / segments_N hashing
Due to additional safety added in #7351 we compute now a strong hash for
.si and segments_N files which are compared during snapshot / restore.
Old snapshots don't have this hash which can cause unnecessary copying
of large amount of data. This commit adds the ability to fetch this
hash from the blob store if needed.

Closes #7434

@s1monw s1monw force-pushed the s1monw:issues/7434 branch to c63626b Aug 26, 2014

@s1monw s1monw merged commit c63626b into elastic:master Aug 26, 2014

@s1monw s1monw deleted the s1monw:issues/7434 branch Aug 26, 2014

@clintongormley clintongormley changed the title [SNAPSHOT] Add BWC layer to .si / segments_N hashing Snapshot/Restore: Add BWC layer to .si / segments_N hashing to identify segments accurately Sep 10, 2014

@clintongormley clintongormley changed the title Snapshot/Restore: Add BWC layer to .si / segments_N hashing to identify segments accurately Resiliency: Add BWC layer to .si / segments_N hashing to identify segments accurately Sep 10, 2014

@clintongormley clintongormley removed the review label Sep 10, 2014

@clintongormley clintongormley added >bug and removed >enhancement labels Sep 26, 2014

@clintongormley clintongormley changed the title Resiliency: Add BWC layer to .si / segments_N hashing to identify segments accurately Add BWC layer to .si / segments_N hashing to identify segments accurately Jun 7, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.