Azure snapshot request returns 500 error: "null_pointer_exception" #28299

tdoman · 2018-01-18T16:11:30Z

NEST 6.0.0-beta1
ES 6.1.1
repository-azure-6.1.1

I am able to successfully create the azure repository but when I attempt to take the snapshot like so:

PUT <server>/_snapshot/es-2018-01/snapshot-2018-01-17?pretty=true&wait_for_completion=false
{
  "include_global_state": true
}

I get the following response:

Response: {
  "error" : {
    "root_cause" : [
      {
        "type" : "null_pointer_exception",
        "reason" : null
      }
    ],
    "type" : "null_pointer_exception",
    "reason" : null
  },
  "status" : 500
}

Here's the log:

[2018-01-17T18:23:58,846][WARN ][r.suppressed             ] path: /_snapshot/es-2018-01/snapshot-2018-01-17, params: {pretty=true, repository=es-2018-01, wait_for_completion=false, snapshot=snapshot-2018-01-17}
java.lang.NullPointerException: null
	at org.elasticsearch.cloud.azure.storage.AzureStorageServiceImpl.generateOperationContext(AzureStorageServiceImpl.java:182) ~[?:?]
	at org.elasticsearch.cloud.azure.storage.AzureStorageServiceImpl.lambda$listBlobsByPrefix$10(AzureStorageServiceImpl.java:312) ~[?:?]
	at org.elasticsearch.cloud.azure.blobstore.util.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:55) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
	at org.elasticsearch.cloud.azure.blobstore.util.SocketAccess.doPrivilegedVoidException(SocketAccess.java:54) ~[?:?]
	at org.elasticsearch.cloud.azure.storage.AzureStorageServiceImpl.listBlobsByPrefix(AzureStorageServiceImpl.java:309) ~[?:?]
	at org.elasticsearch.cloud.azure.blobstore.AzureBlobStore.listBlobsByPrefix(AzureBlobStore.java:118) ~[?:?]
	at org.elasticsearch.cloud.azure.blobstore.AzureBlobContainer.listBlobsByPrefix(AzureBlobContainer.java:123) ~[?:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:769) ~[elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:747) ~[elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:599) ~[elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.snapshots.SnapshotsService.createSnapshot(SnapshotsService.java:236) ~[elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.action.admin.cluster.snapshots.create.TransportCreateSnapshotAction.masterOperation(TransportCreateSnapshotAction.java:82) ~[elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.action.admin.cluster.snapshots.create.TransportCreateSnapshotAction.masterOperation(TransportCreateSnapshotAction.java:41) ~[elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:88) ~[elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:167) ~[elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637) [elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.1.1.jar:6.1.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

The text was updated successfully, but these errors were encountered:

dadoonet · 2018-01-18T16:17:29Z

@tdoman Thanks for reporting. Could you share what you have in elasticsearch.yml (please replace any secret by XXXXX)?
Also is that a fresh new installation? How did you create the repository?

dadoonet · 2018-01-18T16:19:06Z

Looking at the code, I see where this NPE is coming from but it seems to indicate that the Azure client can not be found.

tdoman · 2018-01-18T16:29:25Z

@dadoonet Ok, that'd be a good message in lieu of the NPE but I am able to successfully use the keys, etc. in other tools to access the blob storage. Also, I had been successfully doing this w/ ES 6.0 so I'm not sure what changed. Here's my repository creation request:

PUT _snapshot/es-2018-01?pretty=true
{
  "settings": {
    "container": "elasticsearch-backups",
    "base_path": "es-2018-01",
    "compress": true,
    "chunk_size": "64mb"
  },
  "type": "azure"
}

Here's the elasticsearch.yml config:

cloud:
    azure:
        storage:
            degreed_account:
                account: <my_account>
                key: <my_key>

Oh, and yes it's a fresh installation. I've simply been doing reindex from remote from my ES 1.7.5 cluster but it's not an upgrade of any kind.

dadoonet · 2018-01-18T16:35:21Z

I believe that you are seeing some deprecation notice may be? Anyway, to use the plugin in 6.1, you have to follow the guide here: https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-azure-usage.html

Define the key and secret:

bin/elasticsearch-keystore add azure.client.default.account
bin/elasticsearch-keystore add azure.client.default.key

Remove credentials from elasticsearch.yml.

Then you need to restart the node and you should be good.

Otherwise, you could may be do something like:

_snapshot/es-2018-01?pretty=true
{
  "settings": {
    "client": "degreed_account",
    "container": "elasticsearch-backups",
    "base_path": "es-2018-01",
    "compress": true,
    "chunk_size": "64mb"
  },
  "type": "azure"
}

That "might work" but I'm unsure though.

tdoman · 2018-01-18T17:21:13Z

@dadoonet Ah, ok I see, the configuration methodology changed between 6.0 and 6.1. Ok, I'll try 6.1 approach out today and report back. I did try your "might work" suggestion and it didn't. :) Thank you very much for the prompt response, I really appreciate it.

dadoonet · 2018-01-18T18:07:13Z

Yeah it changed but it is supposed to be backward compatible. That's why I marked this as a bug and also because a NPE is always bad. 😉

tdoman · 2018-01-18T23:46:30Z

@dadoonet Yep, purely configurational, thanks again for the support David.

tdoman · 2018-01-18T23:48:03Z

Sorry, I re-opened it so the NPE can be addressed.

dadoonet · 2018-01-19T05:54:44Z

Thanks for confirming. I'm planning to address the issues indeed (backward comp and NPE). Let's keep that open.

When someone migrates from 5.6 to 6.x with deprecated settings like: ``` cloud: azure: storage: foo: account: <my_account> key: <my_key> ``` It produces a NPE anytime someone wants to use a repository which name is not `default`. This has been introduced by elastic#23518 when I backported it to 6.x branch. In this case, when we generate an OperationContext, we only try to get azure settings from "normal" `storageSettings` with: ```java this.storageSettings.get(clientName) ``` But in the context of deprecated settings, this returns `null` which causes the NPE just after. This commit adds a check and if no settings are found in the "normal" `storageSettings`, we look at settings from `deprecatedStorageSettings`. The reason I missed it in the 7.0 version (master branch) is because actually `deprecatedStorageSettings` has been removed there already. Also I modified the `testGetSelectedClientDefault` method which was only doing exactly the same thing as `testGetDefaultClientWithPrimaryAndSecondaries`. Closes elastic#28299.

* Fix NPE when using deprecated Azure settings When someone migrates from 5.6 to 6.x with deprecated settings like: ``` cloud: azure: storage: foo: account: <my_account> key: <my_key> ``` It produces a NPE anytime someone wants to use a repository which name is not `default`. This has been introduced by #23518 when I backported it to 6.x branch. In this case, when we generate an OperationContext, we only try to get azure settings from "normal" `storageSettings` with: ```java this.storageSettings.get(clientName) ``` But in the context of deprecated settings, this returns `null` which causes the NPE just after. This commit adds a check and if no settings are found in the "normal" `storageSettings`, we look at settings from `deprecatedStorageSettings`. The reason I missed it in the 7.0 version (master branch) is because actually `deprecatedStorageSettings` has been removed there already. Also I renamed the `testGetSelectedClientDefault` method to `testGenerateOperationContext` as it was only doing exactly the same thing as `testGetDefaultClientWithPrimaryAndSecondaries`. Closes #28299.

* Fix NPE when using deprecated Azure settings When someone migrates from 5.6 to 6.x with deprecated settings like: ``` cloud: azure: storage: foo: account: <my_account> key: <my_key> ``` It produces a NPE anytime someone wants to use a repository which name is not `default`. This has been introduced by #23518 when I backported it to 6.x branch. In this case, when we generate an OperationContext, we only try to get azure settings from "normal" `storageSettings` with: ```java this.storageSettings.get(clientName) ``` But in the context of deprecated settings, this returns `null` which causes the NPE just after. This commit adds a check and if no settings are found in the "normal" `storageSettings`, we look at settings from `deprecatedStorageSettings`. The reason I missed it in the 7.0 version (master branch) is because actually `deprecatedStorageSettings` has been removed there already. Also I renamed the `testGetSelectedClientDefault` method to `testGenerateOperationContext` as it was only doing exactly the same thing as `testGetDefaultClientWithPrimaryAndSecondaries`. Closes #28299. Backport of #28769 in 6.2 branch.

ywelsch · 2018-03-27T09:37:47Z

Closed by #28769

dadoonet added >bug :Plugin Repository Azure labels Jan 18, 2018

tdoman mentioned this issue Jan 18, 2018

Azure snapshot request returns 500 error: "null_pointer_exception" elastic/elasticsearch-net#3051

Closed

tdoman closed this as completed Jan 18, 2018

tdoman reopened this Jan 18, 2018

tdoman closed this as completed Jan 18, 2018

tdoman reopened this Jan 18, 2018

dadoonet self-assigned this Jan 19, 2018

clintongormley added :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs and removed :Plugin Repository Azure labels Feb 14, 2018

okke-formsma mentioned this issue Feb 20, 2018

[elastic6] Azure-repository settings should be set using keystore elastic/azure-marketplace#167

Closed

dadoonet mentioned this issue Feb 21, 2018

Fix NPE when using deprecated Azure settings #28769

Merged

ywelsch closed this as completed Mar 27, 2018

brandonmichaelhunter mentioned this issue Sep 27, 2018

Azure snapshot request returns 500 error: #34122

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure snapshot request returns 500 error: "null_pointer_exception" #28299

Azure snapshot request returns 500 error: "null_pointer_exception" #28299

tdoman commented Jan 18, 2018 •

edited

Loading

dadoonet commented Jan 18, 2018

dadoonet commented Jan 18, 2018

tdoman commented Jan 18, 2018 •

edited

Loading

dadoonet commented Jan 18, 2018

tdoman commented Jan 18, 2018

dadoonet commented Jan 18, 2018

tdoman commented Jan 18, 2018

tdoman commented Jan 18, 2018

dadoonet commented Jan 19, 2018

ywelsch commented Mar 27, 2018

Azure snapshot request returns 500 error: "null_pointer_exception" #28299

Azure snapshot request returns 500 error: "null_pointer_exception" #28299

Comments

tdoman commented Jan 18, 2018 • edited Loading

dadoonet commented Jan 18, 2018

dadoonet commented Jan 18, 2018

tdoman commented Jan 18, 2018 • edited Loading

dadoonet commented Jan 18, 2018

tdoman commented Jan 18, 2018

dadoonet commented Jan 18, 2018

tdoman commented Jan 18, 2018

tdoman commented Jan 18, 2018

dadoonet commented Jan 19, 2018

ywelsch commented Mar 27, 2018

tdoman commented Jan 18, 2018 •

edited

Loading

tdoman commented Jan 18, 2018 •

edited

Loading