[CI] S3RepositoryThirdPartyTests testCreateSnapshot failing

[davidkyle]

Build scan:
https://gradle-enterprise.elastic.co/s/pcf7d46eobkok/tests/:plugins:repository-s3:s3ThirdPartyTest/org.elasticsearch.repositories.s3.S3RepositoryThirdPartyTests/testCreateSnapshot

Reproduction line:

./gradlew ':plugins:repository-s3:s3ThirdPartyTest' --tests "org.elasticsearch.repositories.s3.S3RepositoryThirdPartyTests.testCreateSnapshot" -Dtests.seed=6E91002885A322BC -Dtests.locale=el-CY -Dtests.timezone=America/Argentina/Rio_Gallegos -Druntime.java=22

Applicable branches:
8.14, 7.17

Reproduces locally?:
Didn't try

Failure history:
Failure dashboard for org.elasticsearch.repositories.s3.S3RepositoryThirdPartyTests#testCreateSnapshot

Failure excerpt:

org.elasticsearch.repositories.RepositoryVerificationException: [test-repo] path [7.17_third_party_tests_6E91002885A322BC] is not accessible on master node


  Caused by: java.io.IOException: Unable to upload object [7.17_third_party_tests_6E91002885A322BC/tests-01me2U_gREecmL1Oaj9rvg/master.dat] using a single upload

    at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:492)
    at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:128)
    at java.security.AccessController.doPrivileged(AccessController.java:571)
    at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:37)
    at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:126)
    at org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:118)
    at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:263)
    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1688)
    at org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:392)
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.lang.Thread.run(Thread.java:1570)

    Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: ZV990ZWS282RVAXF; S3 Extended Request ID: caWJCNe/MG/jtIgAImkzImp7kPWCfAAWd+jWBWHRl8CH6udOro1hlmLD/ecpoIcWkSIwEDsGQYY=; Proxy: null)

      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
      at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
      at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
      at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
      at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
      at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)
      at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531)
      at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)
      at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)
      at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$19(S3BlobContainer.java:490)
      at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:46)
      at java.security.AccessController.doPrivileged(AccessController.java:319)
      at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:45)
      at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:490)
      at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:128)
      at java.security.AccessController.doPrivileged(AccessController.java:571)
      at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:37)
      at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:126)
      at org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:118)
      at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:263)
      at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1688)
      at org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:392)
      at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777)
      at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
      at java.lang.Thread.run(Thread.java:1570)

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[ywangd]

The error is

The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId;

So not about the test itself but infrastructure related. The test passed locally for me and it has not failed since yesterday. I suspect it is either transient or somehow the underlying 3rd party credentials are updated. I plan to close it in 24 hours if no further failures.

[ywangd]

Unmuted on 8.14 with 25b5660
Will monitor how it goes.

[ywangd]

Unmuted on 7.17 with 42b93a5

[ywangd]

Closing this as explained above.