[ABFS] HADOOP-18656: Paginated Delete Driver Testing #5661
Conversation
saxenapranav
left a comment
saxenapranav
left a comment
There was a problem hiding this comment.
Even during the process of acl-check pagination, server will send X_MS_CONTINUATION?
| import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.*; | ||
| import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.DEFAULT_DELETE_CONSIDERED_IDEMPOTENT; | ||
| import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.SERVER_SIDE_ENCRYPTION_ALGORITHM; | ||
| import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.*; |
| boolean enablePagination = abfsConfiguration.getBoolean( | ||
| ConfigurationKeys.FS_AZURE_ENABLE_PAGINATED_DELETE, | ||
| DEFAULT_ENABLE_PAGINATED_DELETE | ||
| ); | ||
|
|
There was a problem hiding this comment.
lets use abfsConfiguration.isEnabledPaginatedDelete
| for (int i = 1; i <= 515; i++) { | ||
| String dirPath = "/dirLevel1" + String.valueOf(i) + "/dirLevel2" + String.valueOf(i); | ||
| String filePath = rootPath + dirPath + "/file" + String.valueOf(i); | ||
| fs.create(new Path(filePath)); | ||
| } |
There was a problem hiding this comment.
Lets have parallel create.
Why 515?
| private final URL baseUrl; | ||
| private final SharedKeyCredentials sharedKeyCredentials; | ||
| private final String xMsVersion = "2019-12-12"; | ||
| private String xMsVersion = "2019-12-12"; |
There was a problem hiding this comment.
Dont we need to switch it to "2023-08-03"?
Should we switch only for deletePath and not for others?
|
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
2 participants
This PR includes code and test changes for the Pagination feature to be supported for Delete operations. The Pagination feature is to ensure the ACL checks that typically precede a recursive delete call in HNS settings can be conducted in a page-wise manner, leading to lower chances of the operation timing out.
However, to keep with the existing driver test suite, the OSS changes for this on the driver side are intended to include only shorter tests. This is because, testing an end-to-end pagination scenario would require creation and deletion of a non-empty folder with number of resources greater than the current page limit set, which would slow down the entire test runs.
Requirements:
xMs Version: 2023-08-03 and above
New configuration for Paginated delete introduced in this is called
fs.azure.enable.paginated.delete. The default value for this is currently set tofalse. It can be flipped through the above configuration, and the default value can also be flipped totrueif the tenants running the tests support pagination in delete.It is also important to note that the feature is supported only for HNS accounts, where the concept of ACL Checks hold good. Thus, for HNS cases, the pagination parameter and continuation token are coupled together, and a non-null continuation token will be passed in the server response only when the ACL checks are to be conducted for a directory having a greater number of resources than the page size. A non-null continuation token can also be passed by the user in the API call only when pagination behavior is expected.
In the case of FNS accounts, pagination is not supported, and the parameter is ignored even if passed. Here the continuation token refers to what is returned in the server response when the actual number of resources getting listed and deleted are greater than a certain limit, and have no relation to this pagination feature.