Use Azure Bulk Deletes in Azure Repository #53919

original-brownbear · 2020-03-21T15:36:35Z

Now that we upgraded the Azure SDK in #53865 we can start using bulk deletes like we do with S3 and GCS.

Upgrading to 8.6.2 in elastic#53865 broke running against HTTPs endpoints (and hence real azure) because the https url connection needs the newly added permission to work.

Now that we upgraded the Azure SDK to 8.6.2 we can make use of bulk deletes.

elasticmachine · 2020-03-21T15:36:37Z

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

original-brownbear · 2020-03-21T15:38:08Z

...pository-azure/src/main/java/org/elasticsearch/repositories/azure/AzureRepositoryPlugin.java

@@ -80,15 +74,6 @@ AzureStorageService createAzureStoreService(final Settings settings) {
        );
    }

-    @Override
-    public List<ExecutorBuilder<?>> getExecutorBuilders(Settings settings) {


No need to deprecate here. We have never documented this threadpool and it was always seen as an internal setting.
Having it set will not prevent ES from starting up cleanly so I don't see anyone getting hurt here.

original-brownbear · 2020-03-21T15:40:17Z

test/fixtures/azure-fixture/src/main/java/fixture/azure/AzureHttpHandler.java

@@ -67,18 +72,21 @@ public void handle(final HttpExchange exchange) throws IOException {
            assert read == -1 : "Request body should have been empty but saw [" + read + "]";
        }
        try {
+            // Request body is closed in the finally block
+            final BytesReference requestBody = Streams.readFully(Streams.noCloseStream(exchange.getRequestBody()));


Since we needed the request body again I refactored things like we did for the other mocks to full drain the input first thing here as well to keep things simple.

original-brownbear · 2020-03-21T15:40:56Z

plugins/repository-azure/src/main/plugin-metadata/plugin-security.policy

@@ -20,4 +20,5 @@
 grant {
  // azure client opens socket connections for to access repository
  permission java.net.SocketPermission "*", "connect";
+  permission java.lang.RuntimePermission "setFactory";


From #53903 this will go away once #53903 is merged, I just set it to be able to run 3rd party tests locally and verify this actually works now.

tlrx

LGTM, I'm happy to see this gone. I trust your code on the server side logic of the batch request :) I left only minor comments.

tlrx · 2020-03-23T08:40:30Z

...repository-azure/src/main/java/org/elasticsearch/repositories/azure/AzureStorageService.java

+                ++currentBatchSize;
+            } while (blobIterator.hasNext() && currentBatchSize < Constants.BATCH_MAX_REQUESTS);
+            currentBatchSize = 0;
+            logger.trace(() -> new ParameterizedMessage("delete blobs for container [{}], blob [{}]", container, blobs));


In case of more than 256 blobs to delete, this will log the same message multiple times right?

Right :) moved the log statement to the beginning of the method now

tlrx · 2020-03-23T08:42:37Z

...-azure/src/test/java/org/elasticsearch/repositories/azure/AzureBlobStoreRepositoryTests.java

@@ -64,7 +64,7 @@ protected Settings repositorySettings() {

    @Override
    protected Map<String, HttpHandler> createHttpHandlers() {
-        return Collections.singletonMap("/container", new AzureBlobStoreHttpHandler("container"));
+        return Collections.singletonMap("/", new AzureBlobStoreHttpHandler("container"));


Batch request does not prefix with the container name, right?

Jup exactly, that's why I had to expand the context here

tlrx · 2020-03-23T08:43:57Z

test/fixtures/azure-fixture/src/main/java/fixture/azure/AzureHttpHandler.java

+                                "Content-ID: " + contentId++ + " \r\n");
+                            if (blobs.remove(b) == null) {
+                                writer.write("\r\nHTTP/1.1 404 The specified blob does not exist. \r\n" +
+                                    "x-ms-error-code: BlobNotFound \r\n" +


original-brownbear · 2020-03-23T11:22:05Z

Thanks Yannick + Tanguy!

Now that we upgraded the Azure SDK to 8.6.2 in elastic#53865 we can make use of bulk deletes.

Now that we upgraded the Azure SDK to 8.6.2 in #53865 we can make use of bulk deletes.

This reverts commit 23cccf0. Unfortunately SAS token auth still doesn't work with bulk deletes so we can't use them yet. Closes elastic#54080

This reverts commit 23cccf0. Unfortunately SAS token auth still doesn't work with bulk deletes so we can't use them yet. Closes #54080

…lastic#54089) This reverts commit 23cccf0. Unfortunately SAS token auth still doesn't work with bulk deletes so we can't use them yet. Closes elastic#54080

…#54111) This reverts commit 23cccf0. Unfortunately SAS token auth still doesn't work with bulk deletes so we can't use them yet. Closes #54080

…lastic#54089) This reverts commit 23cccf0. Unfortunately SAS token auth still doesn't work with bulk deletes so we can't use them yet. Closes elastic#54080

…#54248) This reverts commit 23cccf0. Unfortunately SAS token auth still doesn't work with bulk deletes so we can't use them yet. Closes #54080

Now that we upgraded the Azure SDK to 8.6.2 in elastic#53865 we can make use of bulk deletes.

…lastic#54089) This reverts commit 23cccf0. Unfortunately SAS token auth still doesn't work with bulk deletes so we can't use them yet. Closes elastic#54080

original-brownbear added 2 commits March 20, 2020 21:33

Fix Azure Repository with HTTPs Endpoint

e04d8d5

Upgrading to 8.6.2 in elastic#53865 broke running against HTTPs endpoints (and hence real azure) because the https url connection needs the newly added permission to work.

Use Azure Repository Bulk Deletes

aa3e01d

Now that we upgraded the Azure SDK to 8.6.2 we can make use of bulk deletes.

original-brownbear added >enhancement :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v8.0.0 v7.7.0 labels Mar 21, 2020

original-brownbear commented Mar 21, 2020

View reviewed changes

shorter diff

1455698

original-brownbear requested review from tlrx and ywelsch March 21, 2020 15:43

tlrx approved these changes Mar 23, 2020

View reviewed changes

ywelsch approved these changes Mar 23, 2020

View reviewed changes

original-brownbear added 2 commits March 23, 2020 11:18

Merge remote-tracking branch 'elastic/master' into azure-bulk-deletes-2

290d325

CR: move log statement

a933c2d

original-brownbear merged commit 23cccf0 into elastic:master Mar 23, 2020

original-brownbear deleted the azure-bulk-deletes-2 branch March 23, 2020 11:22

original-brownbear added a commit to original-brownbear/elasticsearch that referenced this pull request Mar 23, 2020

Use Azure Bulk Deletes in Azure Repository (elastic#53919)

ace1a1b

Now that we upgraded the Azure SDK to 8.6.2 in elastic#53865 we can make use of bulk deletes.

original-brownbear mentioned this pull request Mar 23, 2020

Use Azure Bulk Deletes in Azure Repository (#53919) #53967

Merged

original-brownbear added a commit that referenced this pull request Mar 23, 2020

Use Azure Bulk Deletes in Azure Repository (#53919) (#53967)

b51ea25

Now that we upgraded the Azure SDK to 8.6.2 in #53865 we can make use of bulk deletes.

codebrain mentioned this pull request Apr 1, 2020

7.7.0 meta ticket (Part 2) elastic/elasticsearch-net#4533

Closed

yyff pushed a commit to yyff/elasticsearch that referenced this pull request Apr 17, 2020

Use Azure Bulk Deletes in Azure Repository (elastic#53919)

752b1cf

Now that we upgraded the Azure SDK to 8.6.2 in elastic#53865 we can make use of bulk deletes.

original-brownbear restored the azure-bulk-deletes-2 branch August 6, 2020 18:28

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Azure Bulk Deletes in Azure Repository #53919

Use Azure Bulk Deletes in Azure Repository #53919

original-brownbear commented Mar 21, 2020

elasticmachine commented Mar 21, 2020

original-brownbear Mar 21, 2020

original-brownbear Mar 21, 2020

original-brownbear Mar 21, 2020

tlrx left a comment

tlrx Mar 23, 2020

original-brownbear Mar 23, 2020

tlrx Mar 23, 2020

original-brownbear Mar 23, 2020

tlrx Mar 23, 2020

original-brownbear commented Mar 23, 2020

Use Azure Bulk Deletes in Azure Repository #53919

Use Azure Bulk Deletes in Azure Repository #53919

Conversation

original-brownbear commented Mar 21, 2020

elasticmachine commented Mar 21, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tlrx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

original-brownbear commented Mar 23, 2020