You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Today, when a recursive delete is issued for a large directory in ADLS Gen2 (HNS) account, the directory deletion happens in O(1) but in backend ACL Checks are done recursively for each object inside that directory which in case of large directory could lead to request time out. Pagination is introduced in the Azure Storage Backend for these ACL checks.
More information on how pagination works can be found on public documentation of Azure Delete Path API.
This PR contains changes to support this from client side. To trigger pagination, client needs to add a new query parameter "paginated" and set it to true along with recursive set to true. In return if the directory is large, server might return a continuation token back to the caller. If caller gets back a continuation token, it has to call the delete API again with continuation token along with recursive and pagination set to true. This is similar to directory delete of FNS account.
Pagination is available only in versions "2023-08-03" onwards.
PR also contains functional tests to verify driver works well with different combinations of recursive and pagination features for both HNS and FNS account.
Full E2E testing of pagination requires large dataset to be created and hence not added as part of driver test suite. But extensive E2E testing has been performed.
[ERROR] testCloseOfDataBlockOnAppendComplete(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAppend) Time elapsed: 9.219 s <<< FAILURE!
[ERROR] testListPathWithValueGreaterThanServerMaximum(org.apache.hadoop.fs.azurebfs.ITestAbfsClient) Time elapsed: 226.54 s <<< FAILURE!
[ERROR] testAbfsStreamOps(org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics) Time elapsed: 5.942 s <<< FAILURE!
[ERROR] testExpect100ContinueFailureInAppend(org.apache.hadoop.fs.azurebfs.services.ITestAbfsOutputStream) Time elapsed: 5.002 s <<< ERROR!
[ERROR] testAppendWithChecksumAtDifferentOffsets(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemChecksum) Time elapsed: 6.037 s <<< ERROR!
[ERROR] testTwoWritersCreateAppendNoInfiniteLease(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease) Time elapsed: 3.717 s <<< ERROR!
[ERROR] test_120_terasort(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort) Time elapsed: 4.503 s <<< ERROR!
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
Jira Ticket: https://issues.apache.org/jira/browse/HADOOP-18656
Commit from trunk: 6ed7389
Today, when a recursive delete is issued for a large directory in ADLS Gen2 (HNS) account, the directory deletion happens in O(1) but in backend ACL Checks are done recursively for each object inside that directory which in case of large directory could lead to request time out. Pagination is introduced in the Azure Storage Backend for these ACL checks.
More information on how pagination works can be found on public documentation of Azure Delete Path API.
This PR contains changes to support this from client side. To trigger pagination, client needs to add a new query parameter "paginated" and set it to true along with recursive set to true. In return if the directory is large, server might return a continuation token back to the caller. If caller gets back a continuation token, it has to call the delete API again with continuation token along with recursive and pagination set to true. This is similar to directory delete of FNS account.
Pagination is available only in versions "2023-08-03" onwards.
PR also contains functional tests to verify driver works well with different combinations of recursive and pagination features for both HNS and FNS account.
Full E2E testing of pagination requires large dataset to be created and hence not added as part of driver test suite. But extensive E2E testing has been performed.