Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AccessControlException when restoring backup from azure repository #28662

Closed
okke-formsma opened this issue Feb 13, 2018 · 4 comments
Closed

AccessControlException when restoring backup from azure repository #28662

okke-formsma opened this issue Feb 13, 2018 · 4 comments

Comments

@okke-formsma
Copy link

@okke-formsma okke-formsma commented Feb 13, 2018

Elasticsearch version
Version: 6.2.1, Build: 7299dc3/2018-02-07T19:34:26.990113Z, JVM: 1.8.0_161

Plugins installed: [repository-azure]

JVM version (java -version):
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

OS version (uname -a if on a Unix-like system):
Linux elastic6-dev 4.13.0-32-generic #35-Ubuntu SMP Thu Jan 25 09:13:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
I'm restoring a backup from elasticsearch 5 (or 6) to an elasticsearch 6 machine using the azure-repository plugin. All index definitions seem to be restored properly, but many shards fail (>50%). Some shards are successfully recovered.

This seems similar to a previous issue; #25931

Provide logs (if relevant):

[2018-02-13T14:04:54,847][WARN ][o.e.c.a.s.ShardStateAction] [ela6dev1] [xxx-2017-05][1] received shard failed for shard id [[xxx-2017-05][1]], allocation id [B_vVqxR5SGKfRoSLOZAW9Q], primary term [0], message [failed recovery], failure [RecoveryFailedException[[xxx-2017-05][1]: Recovery failed on {ela6dev1}{6LKL387lRjuE9mDdWWoanQ}{Ip9YutruRCyl7vjIv7M3WA}{10.2.30.4}{10.2.30.4:9300}]; nested: IndexShardRecoveryException[failed recovery]; nested: IndexShardRestoreFailedException[restore failed]; nested: IndexShardRestoreFailedException[failed to restore snapshot [2018-02-03/Pj8IMfHASyOCkl9rv9xO3Q]]; nested: RuntimeException[java.security.AccessControlException: access denied ("java.net.SocketPermission" "xxxelasticbackup.blob.core.windows.net:443" "connect,resolve")]; nested: AccessControlException[access denied ("java.net.SocketPermission" "xxxelasticbackup.blob.core.windows.net:443" "connect,resolve")]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [xxx-2017-05][1]: Recovery failed on {ela6dev1}{6LKL387lRjuE9mDdWWoanQ}{Ip9YutruRCyl7vjIv7M3WA}{10.2.30.4}{10.2.30.4:9300}
        at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$7(IndexShard.java:2065) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:566) [elasticsearch-6.2.1.jar:6.2.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
        at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:342) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:275) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1613) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$7(IndexShard.java:2061) ~[elasticsearch-6.2.1.jar:6.2.1]
        ... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
        at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:463) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:277) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:300) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:275) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1613) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$7(IndexShard.java:2061) ~[elasticsearch-6.2.1.jar:6.2.1]
        ... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [2018-02-03/Pj8IMfHASyOCkl9rv9xO3Q]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:827) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:448) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:277) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:300) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:275) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1613) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$7(IndexShard.java:2061) ~[elasticsearch-6.2.1.jar:6.2.1]
        ... 4 more
Caused by: java.lang.RuntimeException: java.security.AccessControlException: access denied ("java.net.SocketPermission" "xxxelasticbackup.blob.core.windows.net:443" "connect,resolve")
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1506) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3036) ~[?:?]
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:489) ~[?:1.8.0_161]
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347) ~[?:?]
        at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:76) ~[?:?]
        at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:199) ~[?:?]
        at com.microsoft.azure.storage.blob.CloudBlob.downloadRangeInternal(CloudBlob.java:1492) ~[?:?]
        at com.microsoft.azure.storage.blob.BlobInputStream.dispatchRead(BlobInputStream.java:255) ~[?:?]
        at com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:448) ~[?:?]
        at com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420) ~[?:?]
        at org.elasticsearch.index.snapshots.blobstore.SlicedInputStream.read(SlicedInputStream.java:92) ~[elasticsearch-6.2.1.jar:6.2.1]
        at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[?:1.8.0_161]
        at org.elasticsearch.index.snapshots.blobstore.RateLimitingInputStream.read(RateLimitingInputStream.java:69) ~[elasticsearch-6.2.1.jar:6.2.1]
        at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:1.8.0_161]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restoreFile(BlobStoreRepository.java:1572) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restore(BlobStoreRepository.java:1515) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:825) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:448) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:277) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:300) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:275) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1613) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$7(IndexShard.java:2061) ~[elasticsearch-6.2.1.jar:6.2.1]
        ... 4 more
Caused by: java.security.AccessControlException: access denied ("java.net.SocketPermission" "xxxelasticbackup.blob.core.windows.net:443" "connect,resolve")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_161]
        at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_161]
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_161]
        at java.lang.SecurityManager.checkConnect(SecurityManager.java:1051) ~[?:1.8.0_161]
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:541) ~[?:?]
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) ~[?:?]
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) ~[?:?]
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) ~[?:?]
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) ~[?:?]
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[?:1.8.0_161]
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347) ~[?:?]
        at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:119) ~[?:?]
        at com.microsoft.azure.storage.blob.CloudBlob.downloadRangeInternal(CloudBlob.java:1492) ~[?:?]
        at com.microsoft.azure.storage.blob.BlobInputStream.dispatchRead(BlobInputStream.java:255) ~[?:?]
        at com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:448) ~[?:?]
        at com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420) ~[?:?]
        at org.elasticsearch.index.snapshots.blobstore.SlicedInputStream.read(SlicedInputStream.java:92) ~[elasticsearch-6.2.1.jar:6.2.1]
        at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[?:1.8.0_161]
        at org.elasticsearch.index.snapshots.blobstore.RateLimitingInputStream.read(RateLimitingInputStream.java:69) ~[elasticsearch-6.2.1.jar:6.2.1]
        at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:1.8.0_161]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restoreFile(BlobStoreRepository.java:1572) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restore(BlobStoreRepository.java:1515) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:825) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:448) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:277) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:300) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:275) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1613) ~[elasticsearch-6.2.1.jar:6.2.1]
        at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$7(IndexShard.java:2061) ~[elasticsearch-6.2.1.jar:6.2.1]
        ... 4 more
[2018-02-13T14:04:54,920][WARN ][o.e.c.m.MetaDataMappingService] [ela6dev1] [xxx-2017-03] re-syncing mappings with cluster state because of types [[xxx]]
@okke-formsma

This comment has been minimized.

Copy link
Author

@okke-formsma okke-formsma commented Feb 20, 2018

Today I tested the ability to restore backups made by an elastic6 cluster to an elastic6 cluster, and those fail with the same exception.

This is blocking our elastic6 migration. Does anyone have some insight in what is going wrong?

tbrooks8 added a commit to tbrooks8/elasticsearch that referenced this issue Feb 20, 2018
This is related to elastic#28662. It wraps the azure repository inputstream in
an inputstream that ensures `read` calls have socket permissions. This
is because the azure inputstream internally makes service calls.
tbrooks8 added a commit that referenced this issue Feb 21, 2018
This is related to #28662. It wraps the azure repository inputstream in
an inputstream that ensures `read` calls have socket permissions. This
is because the azure inputstream internally makes service calls.
tbrooks8 added a commit to tbrooks8/elasticsearch that referenced this issue Feb 21, 2018
This is related to elastic#28662. It wraps the azure repository inputstream in
an inputstream that ensures `read` calls have socket permissions. This
is because the azure inputstream internally makes service calls.
tbrooks8 added a commit that referenced this issue Feb 21, 2018
This is related to #28662. It wraps the azure repository inputstream in
an inputstream that ensures `read` calls have socket permissions. This
is because the azure inputstream internally makes service calls.
tbrooks8 added a commit that referenced this issue Feb 21, 2018
This is related to #28662. It wraps the azure repository inputstream in
an inputstream that ensures `read` calls have socket permissions. This
is because the azure inputstream internally makes service calls.
@tbrooks8

This comment has been minimized.

Copy link
Contributor

@tbrooks8 tbrooks8 commented Feb 21, 2018

@okke-formsma thank your for opening this issue. It identified a real issue with the azure repository plugin.

A fix has been merged and will be included in 7.0, 6.3, and 6.2.3.

@tbrooks8 tbrooks8 closed this Feb 21, 2018
@okke-formsma

This comment has been minimized.

Copy link
Author

@okke-formsma okke-formsma commented Mar 1, 2018

@tbrooks8 thank you for the fix. Any word on the release date of 6.2.3?

@jasontedor

This comment has been minimized.

Copy link
Member

@jasontedor jasontedor commented Mar 1, 2018

Any word on the release date of 6.2.3?

Sorry, we do not provide release dates.

sebasjm pushed a commit to sebasjm/elasticsearch that referenced this issue Mar 10, 2018
This is related to elastic#28662. It wraps the azure repository inputstream in
an inputstream that ensures `read` calls have socket permissions. This
is because the azure inputstream internally makes service calls.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.