Restore mistake in #1315

[phoebusm]

Reference Issues/PRs

https://github.com/man-group/ArcticDB/pull/1315/files#r1557428013
Batch delete container is needed to be manually cleared after each batch run.
The operation has been incorrectly removed in #1315

What does this implement or fix?

Restore batch delete container clearing operation

Any other comments?

Checklist

Checklist for code changes...

• Have you updated the relevant docstrings, documentation and copyright notice?
• Is this contribution tested against all ArcticDB's features?
• Do all exceptions introduced raise appropriate error messages?
• Are API changes highlighted in the PR description?
• Is the PR labelled as enhancement or bug so it appears in autogenerated release notes?

[phoebusm]

@muhammadhamzasajjad

[poodlewars]

Test?

[phoebusm]

Test?

Will come shortly. This PR is created urgently for fixing the problem as the mistake is critical as it affects all users use Azure as backend storage.
I won't merge the PR until the test is pushed, unless some user does flag it

[muhammadhamzasajjad]

@phoebusm, thanks for raising this. Not clearing to_delete vector makes it significantly slower if we have many batches. The bug must have been affecting the performance or was it also causing errors? AFAIK, azure does silent deletes if you attempt to remove an inexistant blob. How much slower was it?

[phoebusm]

@phoebusm, thanks for raising this. Not clearing to_delete vector makes it significantly slower if we have many batches. The bug must have been affecting the performance or was it also causing errors? AFAIK, azure does silent deletes if you attempt to remove an inexistant blob. How much slower was it?

@muhammadhamzasajjad It is more than just performance problem. Deletion can't be run as exception will be thrown. Or, if delete_library is being called, the exception will be swallowed, which could result in memory leak.

Explation:
If the vector don't get cleared, in the next cycle of the loop, extra blob_name will be added to the vector.
On

ArcticDB/cpp/arcticdb/storage/azure/azure_storage.cpp

Line 222 in 102683d

if (++batch_counter == delete_object_limit) {

, batch_counter will definitely larger than delete_object_limit so all batch delete requests will be run at

ArcticDB/cpp/arcticdb/storage/azure/azure_storage.cpp

Line 229 in 102683d

submit_batch(to_delete);

.
Because of the extra checking you added in

ArcticDB/cpp/arcticdb/storage/azure/azure_real_client.cpp

Line 88 in 102683d

util::check(blob_names.size() <= BATCH_SUBREQUEST_LIMIT,

, exception arcticdb_ext.exceptions.InternalException: E_ASSERTION_FAILURE Azure delete batch size xxx exceeds maximum permitted batch size of 256 will be thrown before the batch requests being submitted to the SDK.

[phoebusm]

@muhammadhamzasajjad Also I have remove the obsolete batch size counter as it becomes obsolete after your refactor.

[phoebusm]

Test?

Just added.