Elasticsearch crashes on startup when upgrading from 8.10.4 to 8.11.1 when S3 snapshots are in use #102173
Assignees
Labels
>bug
:Distributed/Snapshot/Restore
Anything directly related to the `_snapshot/*` APIs
Team:Distributed
Meta label for distributed team
Comments
|
I suspect this is due to #101705, neither of the two suggested issues would explain this. Does it reproduce if you clear |
DaveCTurner
added
:Distributed/Snapshot/Restore
|
Pinging @elastic/es-distributed (Team:Distributed) |
|
@DaveCTurner it no longer reproduces if I set |
|
Thanks @dggreenbaum that confirms the relationship with #101705. |
arteam
added a commit
to arteam/elasticsearch
that referenced
this issue
Nov 15, 2023
Unfortunately, `AWSSecurityTokenServiceClientBuilder#setRegion` is not just a setter on the builder. It looks up the region by its name which laziliy initializes some regional configuration. As a result, the call with an `access denied` error, because the caller doesn't have permission to call `accessDeclaredMembers` in some Jackson internals. We fix that in two ways: * Make sure `withRegion` call is priviliged * Eagarly lookup region metadata in `S3RepositoryPlugin` Fixes elastic#102173
arteam
added a commit
that referenced
this issue
Nov 16, 2023
Unfortunately, `AWSSecurityTokenServiceClientBuilder#setRegion` is not just a setter on the builder. It looks up the region by its name which lazily initializes some regional configuration. As a result, the call with an access denied error, because the caller doesn't have permission to call `accessDeclaredMembers` in some Jackson internals. This bug wasn't caught by the `CustomWebIdentityTokenCredentialsProviderTests#testSupportRegionalizedEndpoints` test because it's under with the test framework that does allow naked reflection calls. We fix that in two ways: * Make sure withRegion call is privileged * Eagerly lookup region metadata in `S3Repository` Fixes #102173
arteam
added a commit
to arteam/elasticsearch
that referenced
this issue
Nov 16, 2023
…c#102230) Unfortunately, `AWSSecurityTokenServiceClientBuilder#setRegion` is not just a setter on the builder. It looks up the region by its name which lazily initializes some regional configuration. As a result, the call with an access denied error, because the caller doesn't have permission to call `accessDeclaredMembers` in some Jackson internals. This bug wasn't caught by the `CustomWebIdentityTokenCredentialsProviderTests#testSupportRegionalizedEndpoints` test because it's under with the test framework that does allow naked reflection calls. We fix that in two ways: * Make sure withRegion call is privileged * Eagerly lookup region metadata in `S3Repository` Fixes elastic#102173
elasticsearchmachine
pushed a commit
that referenced
this issue
Nov 16, 2023
… (#102285) Unfortunately, `AWSSecurityTokenServiceClientBuilder#setRegion` is not just a setter on the builder. It looks up the region by its name which lazily initializes some regional configuration. As a result, the call with an access denied error, because the caller doesn't have permission to call `accessDeclaredMembers` in some Jackson internals. This bug wasn't caught by the `CustomWebIdentityTokenCredentialsProviderTests#testSupportRegionalizedEndpoints` test because it's under with the test framework that does allow naked reflection calls. We fix that in two ways: * Make sure withRegion call is privileged * Eagerly lookup region metadata in `S3Repository` Fixes #102173
rjernst
pushed a commit
to rjernst/elasticsearch
that referenced
this issue
Nov 16, 2023
…c#102230) Unfortunately, `AWSSecurityTokenServiceClientBuilder#setRegion` is not just a setter on the builder. It looks up the region by its name which lazily initializes some regional configuration. As a result, the call with an access denied error, because the caller doesn't have permission to call `accessDeclaredMembers` in some Jackson internals. This bug wasn't caught by the `CustomWebIdentityTokenCredentialsProviderTests#testSupportRegionalizedEndpoints` test because it's under with the test framework that does allow naked reflection calls. We fix that in two ways: * Make sure withRegion call is privileged * Eagerly lookup region metadata in `S3Repository` Fixes elastic#102173
andreidan
pushed a commit
to andreidan/elasticsearch
that referenced
this issue
Nov 22, 2023
…c#102230) Unfortunately, `AWSSecurityTokenServiceClientBuilder#setRegion` is not just a setter on the builder. It looks up the region by its name which lazily initializes some regional configuration. As a result, the call with an access denied error, because the caller doesn't have permission to call `accessDeclaredMembers` in some Jackson internals. This bug wasn't caught by the `CustomWebIdentityTokenCredentialsProviderTests#testSupportRegionalizedEndpoints` test because it's under with the test framework that does allow naked reflection calls. We fix that in two ways: * Make sure withRegion call is privileged * Eagerly lookup region metadata in `S3Repository` Fixes elastic#102173
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Elasticsearch Version
8.11.1
Installed Plugins
No response
Java Version
bundled
OS Version
ubuntu:20.04
Problem Description
I'm orchestrating my Elasticsearch deployment using Cloud on K8s 2.9. When attempting to upgrade my Elasticsearch cluster from 8.10.4 to 8.11.1 the first node to restart crashes with the
access deniedstack trace included in the logs section.This occurs with an unmodified
docker.elastic.co/elasticsearch/elasticsearch:8.11.1image with no additional plugins installed. I am using S3 based snapshots with this cluster. I have been able to successfully do minor version upgrades of this cluster in the past.I suspect this may be related to #101344 or #101245 since they relate to
S3Service.javawhich appears in the stack trace and the patch notes for 8.11.0.Steps to Reproduce
Starting from a functioning ES 8.10.4 cluster with S3 snapshots configured attempt to upgrade an existing data node to 8.11.1. It will fail to start and produce the above stack trace.
Logs (if relevant)
The text was updated successfully, but these errors were encountered: