Deleting files in s3 buckets can trip AccessControlException

[ywangd]

Since 8.13.0, the task runs in the background to delete stale files of snapshots can sometimes run into AccessControlException. The most relevant stacktrace is show below. It's been identified that the issue is with running deletePartition inside the lambda of MapIterator.forEachRemaining which changes the stack frame for SecurityManager and makes the top level doPrivilegedVoid not effective. The lambda version of forEachRemaining was introduced in #103581 which is part of the 8.13.0 release. We should change doPrivileged call to be tightly around the necessary networking call to re-establish correct the permission grant.

Caused by: java.security.AccessControlException: access denied ("java.net.SocketPermission" "x.x.x.x:y" "connect,resolve")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
	at java.security.AccessController.checkPermission(AccessController.java:1085) ~[?:?]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
	at java.lang.SecurityManager.checkConnect(SecurityManager.java:905) ~[?:?]
	at java.net.Socket.connect(Socket.java:748) ~[?:?]
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368) ~[?:?]
	at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:142) ~[?:?]
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[?:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[?:?]
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
	at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) ~[?:?]
	at com.amazonaws.http.conn.$Proxy92.connect(Unknown Source) ~[?:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[?:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[?:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[?:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[?:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[?:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[?:?]
	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1346) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:2335) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.deletePartition(S3BlobStore.java:338) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.lambda$deleteBlobsIgnoringIfNotExists$0(S3BlobStore.java:315) ~[?:?]
	at org.elasticsearch.common.collect.Iterators$MapIterator.lambda$forEachRemaining$0(Iterators.java:175) ~[elasticsearch-8.13.0.jar:?]
	at java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:300) ~[?:?]
	at java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:314) ~[?:?]
	at java.util.Spliterators$1Adapter.forEachRemaining(Spliterators.java:706) ~[?:?]
	at org.elasticsearch.common.collect.Iterators$MapIterator.forEachRemaining(Iterators.java:175) ~[elasticsearch-8.13.0.jar:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.lambda$deleteBlobsIgnoringIfNotExists$1(S3BlobStore.java:312) ~[?:?]
	at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:46) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:45) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.deleteBlobsIgnoringIfNotExists(S3BlobStore.java:311) ~[?:?]
	... 9 more

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)