Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Creating a repository with chunk_size: null causes snapshots to be corrupted #26843

Closed
dakrone opened this issue Sep 29, 2017 · 0 comments

Comments

Projects
None yet
2 participants
@dakrone
Copy link
Member

commented Sep 29, 2017

Performing these operations:

POST /test/doc/1
{"body": "foo"}

# make sure you have path.repo=/tmp
PUT /_snapshot/my_backup
{
  "type": "fs",
  "settings": {
    "chunk_size": null,
    "location": "/tmp/backups"
  }
}

PUT /_snapshot/my_backup/snapshot-1?wait_for_completion=true

# the snapshot will show as successful
GET /_snapshot/my_backup/_all

POST /_snapshot/my_backup/snapshot-1/_restore

On doing the restore, the index will be corrupt and fails with all kinds of:

[2017-09-29T15:32:51,804][WARN ][o.e.c.a.s.ShardStateAction] [zS7gqMd] [test][4] received shard failed for shard id [[test][4]], allocation id [1m3SAW23RiORYN3paf5zRg], primary term [0], message [failed recovery], failure [RecoveryFailedException[[test][4]: Recovery failed on {zS7gqMd}{zS7gqMdkQgenxYl5YYMuPg}{fbrBGXZVSoOiHjueD2o3_Q}{127.0.0.1}{127.0.0.1:9300}]; nested: IndexShardRecoveryException[failed recovery]; nested: IndexShardRestoreFailedException[restore failed]; nested: IndexShardRestoreFailedException[failed to restore snapshot [snapshot-1/bBueY70xTW-w2r3yGoQrmA]]; nested: IndexShardRestoreFailedException[Failed to recover index]; nested: CorruptIndexException[verification failed (hardware problem?) : expected=j36fb4 actual=null footer=null writtenLength=0 expectedLength=197 (resource=name [segments_1], length [197], checksum [j36fb4], writtenBy [6.2.0]) (resource=VerifyingIndexOutput(segments_1))]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [test][4]: Recovery failed on {zS7gqMd}{zS7gqMdkQgenxYl5YYMuPg}{fbrBGXZVSoOiHjueD2o3_Q}{127.0.0.1}{127.0.0.1:9300}
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1710) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:305) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:238) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1317) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1706) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:413) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:240) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:263) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:238) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1317) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1706) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [snapshot-1/bBueY70xTW-w2r3yGoQrmA]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:844) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:408) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:240) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:263) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:238) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1317) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1706) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: Failed to recover index
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restore(BlobStoreRepository.java:1524) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:842) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:408) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:240) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:263) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:238) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1317) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1706) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	... 4 more
Caused by: org.apache.lucene.index.CorruptIndexException: verification failed (hardware problem?) : expected=j36fb4 actual=null footer=null writtenLength=0 expectedLength=197 (resource=name [segments_1], length [197], checksum [j36fb4], writtenBy [6.2.0]) (resource=VerifyingIndexOutput(segments_1))
	at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.verify(Store.java:1124) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.store.Store.verify(Store.java:464) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restoreFile(BlobStoreRepository.java:1586) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restore(BlobStoreRepository.java:1521) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:842) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:408) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:240) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:263) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:238) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1317) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1706) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	... 4 more

@imotov imotov self-assigned this Sep 29, 2017

imotov added a commit to imotov/elasticsearch that referenced this issue Sep 30, 2017

Snapshot/Restore: better handle incorrect chunk_size settings in FS repo
Specifying a negative value or null as a chunk_size in FS repository can lead to corrupt snapshots.

Closes elastic#26843

@imotov imotov closed this in #26844 Nov 8, 2017

imotov added a commit that referenced this issue Nov 8, 2017

Snapshot/Restore: better handle incorrect chunk_size settings in FS r…
…epo (#26844)

Specifying a negative value or null as a chunk_size in FS repository can lead to corrupt snapshots.

Closes #26843

imotov added a commit that referenced this issue Nov 8, 2017

Snapshot/Restore: better handle incorrect chunk_size settings in FS r…
…epo (#26844)

Specifying a negative value or null as a chunk_size in FS repository can lead to corrupt snapshots.

Closes #26843

imotov added a commit that referenced this issue Nov 8, 2017

Snapshot/Restore: better handle incorrect chunk_size settings in FS r…
…epo (#26844)

Specifying a negative value or null as a chunk_size in FS repository can lead to corrupt snapshots.

Closes #26843

imotov added a commit that referenced this issue Nov 8, 2017

Snapshot/Restore: better handle incorrect chunk_size settings in FS r…
…epo (#26844)

Specifying a negative value or null as a chunk_size in FS repository can lead to corrupt snapshots.

Closes #26843
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.