Losing working secure settings of s3-repository client during runtime

[tiwisag]

Elasticsearch Version

8.17.4

Installed Plugins

repository-url,rest-root,x-pack-core,x-pack-redact,ingest-user-agent,x-pack-async-search,x-pack-monitoring, repository-s3, x-pack-analytics, x-pack-esql-core, x-pack-ent-search, x-pack-autoscaling, lang-painless, x-pack-ml, lang-mustache, legacy-geo, logsdb, x-pack-ql, rank-rrf, analysis-common, health-shards-availability, transport-netty4, aggregations, ingest-common, frozen-indices, x-pack-identity-provider, x-pack-shutdown, x-pack-text-structure, snapshot-repo-test-kit, ml-package-loader, kibana, constant-keyword, x-pack-logstash, x-pack-ccr, x-pack-graph, x-pack-esql, parent-join, counted-keyword, x-pack-enrich, repositories-metering-api, transform, repository-azure, dot-prefix-validation, repository-gcs, spatial, x-pack-otel-data, apm, mapper-extras, mapper-version, x-pack-rollup, percolator, data-streams, x-pack-stack, rank-eval, reindex, x-pack-security, blob-cache, searchable-snapshots, x-pack-slm, x-pack-geoip-enterprise-downloader, snapshot-based-recoveries, x-pack-watcher, old-lucene-versions, x-pack-ilm, x-pack-inference, x-pack-voting-only-node, x-pack-deprecation, x-pack-fleet, x-pack-aggregate-metric, x-pack-downsample, x-pack-profiling, ingest-geoip, x-pack-write-load-forecaster, search-business-rules, ingest-attachment, wildcard, x-pack-apm-data, unsigned-long, x-pack-sql, runtime-fields-common, x-pack-async, vector-tile, lang-expression, x-pack-eql, prometheus-exporter, mining-es-plugin

Java Version

21.0.7

OS Version

Alpine Linux v3.21

Problem Description

When the elasticsearch cluster is running, after some time it seems to lose the secure client settings although they are still set in the keystore.

The cluster is running on EKS and consists of multiple nodes. The s3-repository plugin of the elasticsearch is used to write/backup/restore indices into s3. For the s3 configuration the secure client settings (s3.client.backup.access_key, s3.client.backup.secret_key, s3.client.restore.access_key, s3.client.restore.secret_key) are saved in the elasticsearch keystore, the others are added with the corresponding api.

The repository verification is acknowleged and backups/restore is working at the beginning. After some time the "Unknown s3 client name [backup]. Existing client configs: default" exception is thrown.

Steps to Reproduce

1. Start elasticsearch

2. Add the secure settings in the Keystore (the files come from Kubernetes Secrets mounted in the pod)
   elasticsearch-keystore add-file -f s3.client.backup.account /objectStoreCredentials/backup.account/backup.account elasticsearch-keystore add-file -f s3.client.backup.key /objectStoreCredentials/backup.key/backup.key elasticsearch-keystore add-file -f s3.client.restore.account /objectStoreCredentials/restore.account/restore.account elasticsearch-keystore add-file -f s3.client.restore.key /objectStoreCredentials/restore.key/restore.key

3. Validate their existence
   elasticsearch-default-2@/#elasticsearch-keystore list warning: ignoring JAVA_HOME=/opt/jre; using ES_JAVA_HOME bootstrap.password keystore.seed s3.client.backup.access_key s3.client.backup.secret_key s3.client.restore.access_key s3.client.restore.secret_key

4. reload secure settings on all nodes
   curl -s -H "$AUTH_HEADER" -X POST "localhost:14220/_nodes/elasticsearch-default-2/reload_secure_settings"
   Check if the number of failed nodes in the response is 0
   { "_nodes": { "total": 1, "successful": 1, "failed": 0 }, "cluster_name": "our_cluster", "nodes": { "pQHNt5rXTTWNvUgOrdynKg": { "name": "elasticsearch-default-2" } } }

5. Create repository
   -------------Request with Body----------------
   curl -s -H "$AUTH_HEADER" -X PUT"localhost:14220/__snapshot/tiwi"
{ "type": "s3", "settings": { "bucket": "backups", "container": "backups", "compress": "true", "client": "backup", "base_path": "backups/test/esrepositories", "region": "eu-north-1" } }
   -------------Response------------
   { "acknowledged": true }

6. Validate repository
   ------ Request ----------
   curl -s -H "$AUTH_HEADER" -X POST "localhost:14220/__snapshot/tiwi/verify"
   --------Response-----------
   { "nodes": { "3JGGPzTcTqOLE15dD1lHfg": { "name": "elasticsearch-default-3" }, "_DL2jNuUSReoHWpeWHykAw": { "name": "elasticsearch-default-0" }, "bHreZzJYRTmJWzC0-M9pyA": { "name": "elasticsearch-default-1" }, "mAqhqrR2Szq4qkDM0GfisA": { "name": "elasticsearch-default-5" }, "ZZtB4G0rT0CQEaTj2ZWWzA": { "name": "elasticsearch-default-4" }, "PsP6zj6gQR-1IKrJ8h3nDQ": { "name": "elasticsearch-default-2" } } }

7. After some time (no restarts), when triggering snapshots:
   { "error": { "root_cause": [ { "type": "repository_verification_exception", "reason": "[tiwi] [[ZZtB4G0rT0CQEaTj2ZWWzA, 'org.elasticsearch.transport.RemoteTransportException: [elasticsearch-default-4][10.0.123.232:14230][internal:admin/repository/verify]'], [PsP6zj6gQR-1IKrJ8h3nDQ, 'org.elasticsearch.transport.RemoteTransportException: [elasticsearch-default-2][10.0.225.15:14230][internal:admin/repository/verify]']]", "suppressed": [ { "type": "repository_verification_exception", "reason": "[tiwi] store location [backups] is not accessible on the node [{elasticsearch-default-4}{ZZtB4G0rT0CQEaTj2ZWWzA}{XLQFDCB0TC6tV7dOg7y4Rg}{elasticsearch-default-4}{10.0.123.232}{10.0.123.232:14230}{cdfhilmrstw}{8.17.4}{7000099-8521000}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true, zone=eu-north-1a}]" }, { "type": "repository_verification_exception", "reason": "[tiwi] store location [backups] is not accessible on the node [{elasticsearch-default-2}{PsP6zj6gQR-1IKrJ8h3nDQ}{3ppI5XomQ76K51DaFNyr5g}{elasticsearch-default-2}{10.0.225.15}{10.0.225.15:14230}{cdfhilmrstw}{8.17.4}{7000099-8521000}{transform.config_version=10.0.0, xpack.installed=true, zone=eu-north-1c, ml.config_version=12.0.0}]" } ] } ], "type": "repository_verification_exception", "reason": "[tiwi] [[ZZtB4G0rT0CQEaTj2ZWWzA, 'org.elasticsearch.transport.RemoteTransportException: [elasticsearch-default-4][10.0.123.232:14230][internal:admin/repository/verify]'], [PsP6zj6gQR-1IKrJ8h3nDQ, 'org.elasticsearch.transport.RemoteTransportException: [elasticsearch-default-2][10.0.225.15:14230][internal:admin/repository/verify]']]", "suppressed": [ { "type": "repository_verification_exception", "reason": "[tiwi] store location [backups] is not accessible on the node [{elasticsearch-default-4}{ZZtB4G0rT0CQEaTj2ZWWzA}{XLQFDCB0TC6tV7dOg7y4Rg}{elasticsearch-default-4}{10.0.123.232}{10.0.123.232:14230}{cdfhilmrstw}{8.17.4}{7000099-8521000}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true, zone=eu-north-1a}]", "caused_by": { "type": "illegal_argument_exception", "reason": "Unknown s3 client name [backup]. Existing client configs: default" } }, { "type": "repository_verification_exception", "reason": "[tiwi] store location [backups] is not accessible on the node [{elasticsearch-default-2}{PsP6zj6gQR-1IKrJ8h3nDQ}{3ppI5XomQ76K51DaFNyr5g}{elasticsearch-default-2}{10.0.225.15}{10.0.225.15:14230}{cdfhilmrstw}{8.17.4}{7000099-8521000}{transform.config_version=10.0.0, xpack.installed=true, zone=eu-north-1c, ml.config_version=12.0.0}]", "caused_by": { "type": "illegal_argument_exception", "reason": "Unknown s3 client name [backup]. Existing client configs: default" } } ] }, "status": 500 }

Logs (if relevant)

No response

[DaveCTurner]

What does elasticsearch-keystore list report after the snapshots have started failing?

Also, does it still fail like this if you set a s3.client.backup.${SOMETHING} setting in elasticsearch.yml? For instance, try setting s3.client.backup.region: eu-north-1.

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[tiwisag]

Hi @DaveCTurner , thx for responding.

It still has all 4 settings in the keystore, so its still there.

If i now retrigger reload secure settings on this node, it works until the exception appears again.

So you mean, adding a setting for the client backup in the elasticsearch.yml which isn't a secure setting so that at least something is set for this client in the "not" secure settings?

I can try to set this up.

[DaveCTurner]

If i know retrigger reload secure Settings on this node, it works until the exception appears again.

Thanks ok, that is odd. Could you set logger.org.elasticsearch.action.admin.cluster.node.reload.TransportNodesReloadSecureSettingsAction: DEBUG? This should report every time ES reloads the keystore in the logs. I can't see a way for it to see these settings change without such a reload.

My suspicion is that k8s is doing something unexpected to the mounted secret and triggering the reload API itself.

[DaveCTurner]

So you mean, adding a setting for the client backup in the elasticsearch.yml which isn't a secure setting so that at least something is set for this client in the "not" secure settings?

Yes that's right.

[tiwisag]

Okay, i will add both settings and will get back to you!

[tiwisag]

elasticsearch.txt

So thats the current adapted config. I will start testing again, and keep an eye on the results over the weekend.

[tiwisag]

Update!
So since the changes which i made from your suggestions on Friday, i was not able to observe this "Unknown s3 client name [backup]. Existing client configs: default" Exception again.

Today and yesterday morning the backups and s3 uploads went smoothly, unfortunately after that the following backups ( tested several times) i retrieve mostly the read time out exception and sometimes this gateway time out exception (both attached as files):

read_timeout.txt
gateway_timeout.txt

My next step is going to get in contact with aws regarding those, but would be interested if you saw similar behaviour already and keep you updated.

[DaveCTurner]

Ah ok no this is the same problem still, Elasticsearch thinks it has no credentials for this client in the keystore so it's falling back to InstanceMetadataServiceCredentialsFetcher which tries to get them from IMDS - this isn't available within these containers so the connection to IMDS is timing out.

Can you share complete logs, from a point where a snapshot worked, past the point where it failed like this, then past the point when you reloaded the credentials and up to the point when the snapshots started working again?