Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure repository creation Issue in Elasticsearch 2.4.0 #20633

Closed
GaneshbabuRamamoorthy opened this issue Sep 22, 2016 · 13 comments
Closed

Azure repository creation Issue in Elasticsearch 2.4.0 #20633

GaneshbabuRamamoorthy opened this issue Sep 22, 2016 · 13 comments

Comments

@GaneshbabuRamamoorthy
Copy link

GaneshbabuRamamoorthy commented Sep 22, 2016

Hi All,

I want to take snapshot and restore using azure repository in Linux, I have configured these below setting in yml,

##################### Azure cloud plugin for repository ######################

cloud:
  azure:
    storage:
      account: "https://XXX.blob.core.windows.net/"
      key: "..."

with the above setting, service is started, but i am m getting below error when creating repository in Elasticsearch 2.4.0,

PUT snapshot/mybackup
{
"type": "azure"
}

Error msg:-

{
   "error": {
      "root_cause": [
         {
            "type": "repository_exception",
            "reason": "[elasticsearch-snapshots] Can not find an azure client for account [null]"
         }
      ],
      "type": "repository_exception",
      "reason": "[elasticsearch-snapshots] Can not find an azure client for account [null]"
   },
   "status": 500
}

When I looked into the ES logs I could see cloud storage account not getting created at the start of elasticsearch

2016-09-22 07:30:23,221][INFO ][http                     ] [mue2rhegrdd002_RD_NDX_MASTER] publish_address {10.247.0.4:9200}, bound_addresses {10.247.0.4:9200}
[2016-09-22 07:30:23,221][INFO ][node                     ] [mue2rhegrdd002_RD_NDX_MASTER] started
[2016-09-22 07:30:23,258][INFO ][cluster.service          ] [mue2rhegrdd002_RD_NDX_MASTER] new_master {mue2rhegrdd002_RD_NDX_MASTER}{-mRr-uLKQkyEJwfqLLI_2Q}{10.247.0.4}{10.247.0.4:9300}{data=false, max_local_storage_nodes=1, master=true}, added {{mue2rhegrdd001_RD_NDX_DATA}{O-JSVOlMQXCuwb5nf5XheQ}{10.247.0.5}{10.247.0.5:9260}{max_local_storage_nodes=1, master=false},{mue2rhegrdd001_RD_NDX_MASTER}{DMAPO7O6QEuXmcsP6aGuAQ}{10.247.0.5}{10.247.0.5:9300}{data=false, max_local_storage_nodes=1, master=true},}, reason: zen-disco-join(elected_as_master, [1] joins received)
**[2016-09-22 07:30:24,576][ERROR][cloud.azure.storage      ] [mue2rhegrdd002_RD_NDX_MASTER] can not create azure storage client: Primary and secondary location URIs in a StorageUri must point to the same resource.**
[2016-09-22 07:30:24,589][INFO ][watcher                  ] [mue2rhegrdd002_RD_NDX_MASTER] starting watch service...
[2016-09-22 07:30:24,590][INFO ][watcher                  ] [mue2rhegrdd002_RD_NDX_MASTER] watch service has started
[2016-09-22 07:30:24,631][INFO ][license.plugin.core      ] [mue2rhegrdd002_RD_NDX_MASTER] license [cf8a2152-4f41-4739-8459-4137cac8105b] - valid
[2016-09-22 07:30:25,489][INFO ][gateway                  ] [mue2rhegrdd002_RD_NDX_MASTER] recovered [20] indices into cluster_state
[2016-09-22 07:30:26,021][INFO ][http                     ] [mue2rhegrdd002_RD_NDX_DATA] publish_address {10.247.0.4:9240}, bound_addresses {10.247.0.4:9240}
[2016-09-22 07:30:26,021][INFO ][node                     ] [mue2rhegrdd002_RD_NDX_DATA] started
[2016-09-22 07:30:26,622][INFO ][cluster.service          ] [mue2rhegrdd002_RD_NDX_MASTER] added {{mue2rhegrdd002_RD_NDX_DATA}{eE1n1iLMQom6VEgYfBwJ9g}{10.247.0.4}{10.247.0.4:9260}{max_local_storage_nodes=1, master=false},}, reason: zen-disco-join(join from node[{mue2rhegrdd002_RD_NDX_DATA}{eE1n1iLMQom6VEgYfBwJ9g}{10.247.0.4}{10.247.0.4:9260}{max_local_storage_nodes=1, master=false}])
[2016-09-22 07:30:26,678][INFO ][cluster.service          ] [mue2rhegrdd002_RD_NDX_DATA] detected_master {mue2rhegrdd002_RD_NDX_MASTER}{-mRr-uLKQkyEJwfqLLI_2Q}{10.247.0.4}{10.247.0.4:9300}{data=false, max_local_storage_nodes=1, master=true}, added {{mue2rhegrdd001_RD_NDX_DATA}{O-JSVOlMQXCuwb5nf5XheQ}{10.247.0.5}{10.247.0.5:9260}{max_local_storage_nodes=1, master=false},{mue2rhegrdd001_RD_NDX_MASTER}{DMAPO7O6QEuXmcsP6aGuAQ}{10.247.0.5}{10.247.0.5:9300}{data=false, max_local_storage_nodes=1, master=true},{mue2rhegrdd002_RD_NDX_MASTER}{-mRr-uLKQkyEJwfqLLI_2Q}{10.247.0.4}{10.247.0.4:9300}{data=false, max_local_storage_nodes=1, master=true},}, reason: zen-disco-receive(from master [{mue2rhegrdd002_RD_NDX_MASTER}{-mRr-uLKQkyEJwfqLLI_2Q}{10.247.0.4}{10.247.0.4:9300}{data=false, max_local_storage_nodes=1, master=true}])
**[2016-09-22 07:30:26,711][ERROR][cloud.azure.storage      ] [mue2rhegrdd002_RD_NDX_DATA] can not create azure storage client: Primary and secondary location URIs in a StorageUri must point to the same resource.**
[2016-09-22 07:30:26,764][INFO ][license.plugin.core      ] [mue2rhegrdd002_RD_NDX_DATA] license [cf8a2152-4f41-4739-8459-4137cac8105b] - valid

Please let me know your suggestions to resolve this.

Thanks,
Ganeshbabu R

@tlrx
Copy link
Member

tlrx commented Sep 22, 2016

@GaneshbabuRamamoorthy I edited your message to remove sensitive information. I advise you to change your credentials as soon as possible.

@dadoonet
Copy link
Member

You are missing one level in your settings.

cloud:
  azure:
    storage:
        my_account:
            account: your_azure_storage_account
            key: your_azure_storage_key

See https://www.elastic.co/guide/en/elasticsearch/plugins/current/cloud-azure-repository.html

@GaneshbabuRamamoorthy
Copy link
Author

@tlrx Thanks sure will changes the credentials.

@dadoonet
As per your recommendation I changed the yml file in all the nodes and then restarted each nodes when I checked in the logs again I could see the same error,

[ERROR][cloud.azure.storage] [localhost_RD_NDX_DATA] can not create azure storage client: Primary and secondary location URIs in a StorageUri must point to the same resource.

Please find the attached yml file and error log and let me know your response.

Thanks,
Ganeshbabu R

YML and Error files.zip

@gmoskovicz
Copy link
Contributor

@dadoonet Can you please double check this again?

WIth the following looks like failing as well

cloud:
  azure:
    storage:
        my_account:
            account: "xxxxx...."
            key: "xxxxx...."

And getting the following error:

[2016-09-26 03:51:43,720][WARN ][rest.suppressed          ] path: /_snapshot/myrepository, params: {repository=myrepository}
RemoteTransportException[[zzz][xxxxx][cluster:admin/repository/put]]; nested: RepositoryException[[elasticsearch-snapshots] Can not find an azure client for account [null]];
Caused by: RepositoryException[[elasticsearch-snapshots] Can not find an azure client for account [null]]
    at org.elasticsearch.cloud.azure.storage.AzureStorageServiceImpl.createContainer(AzureStorageServiceImpl.java:168)
    at org.elasticsearch.cloud.azure.blobstore.AzureBlobStore.createContainer(AzureBlobStore.java:117)
    at org.elasticsearch.repositories.azure.AzureRepository.startVerification(AzureRepository.java:179)
    at org.elasticsearch.repositories.RepositoriesService.verifyRepository(RepositoriesService.java:211)
    at org.elasticsearch.repositories.RepositoriesService$VerifyingRegisterRepositoryListener.onResponse(RepositoriesService.java:436)
    at org.elasticsearch.repositories.RepositoriesService$VerifyingRegisterRepositoryListener.onResponse(RepositoriesService.java:421)
    at org.elasticsearch.cluster.AckedClusterStateUpdateTask.onAllNodesAcked(AckedClusterStateUpdateTask.java:63)
    at org.elasticsearch.cluster.service.InternalClusterService$SafeAckedClusterStateTaskListener.onAllNodesAcked(InternalClusterService.java:733)
    at org.elasticsearch.cluster.service.InternalClusterService$AckCountDownListener.onNodeAck(InternalClusterService.java:1013)
    at org.elasticsearch.cluster.service.InternalClusterService$DelegetingAckListener.onNodeAck(InternalClusterService.java:952)
    at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:637)
    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

@gmoskovicz gmoskovicz reopened this Sep 26, 2016
@dadoonet
Copy link
Member

@gmoskovicz I can't reproduce it so I probably need more details. Here is what I did so far:

On a fresh new and clean 2.4.0 installation:

bin/plugin install cloud-azure
vi config/elasticsearch.yml 
cloud:
    azure:
        storage:
            my_account:
                account: MY_ACCOUNT
                key: MY_KEY
bin/elasticsearch

Then:

curl -XDELETE 127.0.0.1:9200/_snapshot/azure?pretty
curl -XPUT '127.0.0.1:9200/_snapshot/azure?pretty' -d '{
  "type": "azure"
}'
curl -XDELETE 127.0.0.1:9200/foo?pretty
curl -XPUT 127.0.0.1:9200/foo/doc/1?pretty -d '{ "foo": "bar" }'
curl -XPUT "127.0.0.1:9200/_snapshot/azure/snapshot_1?wait_for_completion=true&pretty"

Everything went well.

Can you tell me exactly what is the command used to create the repository?
Was that an existing repository for a previous elasticsearch version and we upgrade the version by any chance?

@GaneshbabuRamamoorthy
Copy link
Author

GaneshbabuRamamoorthy commented Sep 27, 2016

Hi @dadoonet,

I modified the below changes in all yml files,

cloud:
  azure:
    storage:
        my_account:
            account: "xxxxxxxxxxx"
            key: "xxxxxxxxxxxxxxxxxxx"

then I started the ES and below is the error msg I am getting in logs,
[2016-09-27 08:25:55,633][ERROR][cloud.azure.storage] [xxxxxxx] can not create azure storage client: Primary and secondary location URIs in a StorageUri must point to the same resource.

when I tried to create the repository using sense,

PUT _snapshot/mybackup
{
"type": "azure",
"settings": {
"account": "my_account"
}
}

I am getting the following error msg,

{
"error": {
"root_cause": [
{
"type": "repository_exception",
"reason": "[elasticsearch-snapshots] Can not find an azure client for account [my_account]"
}
],
"type": "repository_exception",
"reason": "[elasticsearch-snapshots] Can not find an azure client for account [my_account]"
},
"status": 500
}

Please confirm me is the cloud azure yml configuration is correct?

Let me know if you need further informations.

Thanks

@dadoonet
Copy link
Member

Ok. So I can reproduce it actually. It sounds like to happen when your key is incorrect.
If your key is correct, then it works just fine.

So to me here, we are swallowing the original exception which is far from being ideal. I believe we should raise a better exception.

So for now, you have to check in the azure console that your credentials are correct.

I'm using here for example the key1.

access keys - microsoft azure google chrome aujourd hui at 15 22 18

@dadoonet
Copy link
Member

BTW I'm pretty sure you are seeing a similar trace as this one, right?

[2016-09-27 15:19:55,151][WARN ][rest.suppressed          ] path: /_snapshot/azure, params: {pretty=, repository=azure}
RepositoryException[[elasticsearch-snapshots] Can not find an azure client for account [my_account]]
    at org.elasticsearch.cloud.azure.storage.AzureStorageServiceImpl.createContainer(AzureStorageServiceImpl.java:168)
    at org.elasticsearch.cloud.azure.blobstore.AzureBlobStore.createContainer(AzureBlobStore.java:117)
    at org.elasticsearch.repositories.azure.AzureRepository.startVerification(AzureRepository.java:179)
    at org.elasticsearch.repositories.RepositoriesService.verifyRepository(RepositoriesService.java:211)
    at org.elasticsearch.repositories.RepositoriesService$VerifyingRegisterRepositoryListener.onResponse(RepositoriesService.java:436)
    at org.elasticsearch.repositories.RepositoriesService$VerifyingRegisterRepositoryListener.onResponse(RepositoriesService.java:421)
    at org.elasticsearch.cluster.AckedClusterStateUpdateTask.onAllNodesAcked(AckedClusterStateUpdateTask.java:63)
    at org.elasticsearch.cluster.service.InternalClusterService$SafeAckedClusterStateTaskListener.onAllNodesAcked(InternalClusterService.java:733)
    at org.elasticsearch.cluster.service.InternalClusterService$AckCountDownListener.onNodeAck(InternalClusterService.java:1013)
    at org.elasticsearch.cluster.service.InternalClusterService$DelegetingAckListener.onNodeAck(InternalClusterService.java:952)
    at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:637)
    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

@dadoonet
Copy link
Member

If you want to print the detailed logs of what is happening, you can also change your logging level with:

curl -XPUT '127.0.0.1:9200/_cluster/settings?pretty' -d '{
  "transient": {
    "logger.cloud.azure": "TRACE"
  }
}'

But sadly, there is a bug in the TRACE log and we are not printing the exception message! :(

logger.trace("fails creating container [{}]", container, e.getMessage());

Should be something like:

logger.trace("fails creating container [{}]", e, container);

@GaneshbabuRamamoorthy
Copy link
Author

GaneshbabuRamamoorthy commented Sep 27, 2016

Yes, Its the same trace I also got in the logs..

When I asked my infrastructure team to create repository in Azure cloud machine and they replied back with the following account and key details.

account: https://devesrepo.blob.core.windows.net/
key: SECRET

Seems the issue is based on Account name, I have given the details as such in the yml on all the nodes, when I modified the yml of account name as "devesrepo" on all nodes and then I restarted the Elasticsearch, I couldn't see any error msg's, now I am able to create repository.

Regards,
Ganeshbabu R

@dadoonet
Copy link
Member

Oh I see. The account is in fact devesrepo.

PLEASE ask your team to regenerate all the credentials. You exposed here sensitive information.

@dadoonet
Copy link
Member

I'm going to leave your issue opened as I'd like to fix at least the logs.

dadoonet added a commit to dadoonet/elasticsearch that referenced this issue Sep 27, 2016
This commit fixes 2 things.

First, we were swallowing the original exception when creating a client with bad credentials.
So even in `TRACE` log level, nothing useful were coming out of it.
With this commit, it nows prints:

```
[2016-09-27 15:54:13,118][ERROR][cloud.azure.storage      ] [node_s0] can not create azure storage client: Storage Key is not a valid base64 encoded string.
```

Closes elastic#20633.

It also fixes running integration tests manually which were not working and were failing with a NPE when starting.
dadoonet added a commit to dadoonet/elasticsearch that referenced this issue Sep 27, 2016
We were swallowing the original exception when creating a client with bad credentials.
So even in `TRACE` log level, nothing useful were coming out of it.
With this commit, it now prints:

```
[2016-09-27 15:54:13,118][ERROR][cloud.azure.storage      ] [node_s0] can not create azure storage client: Storage Key is not a valid base64 encoded string.
```

Closes elastic#20633.

Backport of elastic#20669 for master branch (6.0)
dadoonet added a commit that referenced this issue Oct 6, 2016
We were swallowing the original exception when creating a client with bad credentials.
So even in `TRACE` log level, nothing useful were coming out of it.
With this commit, it now prints:

```
[2016-09-27 15:54:13,118][ERROR][cloud.azure.storage      ] [node_s0] can not create azure storage client: Storage Key is not a valid base64 encoded string.
```

Closes #20633.

Backport of #20669 for 5.x branch (5.1)

(cherry picked from commit 14af343)
dadoonet added a commit that referenced this issue Oct 6, 2016
We were swallowing the original exception when creating a client with bad credentials.
So even in `TRACE` log level, nothing useful were coming out of it.
With this commit, it now prints:

```
[2016-09-27 15:54:13,118][ERROR][cloud.azure.storage      ] [node_s0] can not create azure storage client: Storage Key is not a valid base64 encoded string.
```

Closes #20633.

Backport of #20669 for 5.0 branch

(cherry picked from commit 14af343)
(cherry picked from commit 6345640)
@khosrojerdi
Copy link

khosrojerdi commented Aug 27, 2018

dear @dadoonet,
I know it is quite late to put a comment here but I need help in same area.
For using the azure as my repository, I need to use https://.blob.core.cloudapi.de/ instead of "https://.blob.core.windows.net/" but I could not find any where to set this host!
I am using elastic search version 5.5.1.

Right now, I am getting the following error:
java.net.UnknownHostException: ***.blob.core.windows.net
...

Thanks in advance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants