azureblob: allow working with only list+create/write permissions (fix no_head_object/add no_read_for_metadata) #7813
Conversation
… no_head_object/add no_read_for_metadata) There are three reasons an option why one would avoid HEAD requests on azureblob, with this now mainly implementing the latter one: - no_head_object: Don't do a HEAD *before* writing to (or reading from) an object, to reduce the number of transactions and thus increase performance. **This already exists in azureblob, a small issue was fixed here where using a prefix would still require a HEAD request to the prefix itself.** - no_head: Don't do a HEAD *after* writing to an object, for the same reason but at the cost of "increasing the chance for undetected upload failures". **This is still not implemented for azureblob yet and quite complex, see s3.Object.Update()!** - no_read_for_metadata: Instead of doing a HEAD to get metadata, use a list operation, to avoid requiring READ permissions for immutable/append-only applications (this seems to be quite azureblob-specific) - one could also use both no_head_object and no_head together to achieve this, but then uploads wouldn't be verified. Fixes rclone#6162 and rclone#7027
| } | ||
| } | ||
|
|
||
| // readMetaData gets the metadata using a ListBlobsFlatPager, to avoid requiring read access on the blob itself, and is used by Fs.readMetaData if no_read_for_metadata is set |
There was a problem hiding this comment.
We could be setting MaxResults in container.ListBlobsFlatOptions to limit it to 1 result which might be a good idea. What do you think?
There was a problem hiding this comment.
The issue is if there's a file hello and a file helloworld.txt, there's a chance that requesting the metadata for hello returns the metadata for helloworld.txt first (as the former is a prefix of the latter).
There was a problem hiding this comment.
Ah wait, there's this:
Blob names are returned in lexicographic order. For more information, see https://docs.microsoft.com/rest/api/storageservices/list-blobs.
(from https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob#Client.NewListBlobsFlatPager)
So, limiting to 1 result should be possible.
There was a problem hiding this comment.
If we can limit it, that would be good.
|
|
||
| for _, currentBlob := range blobs.ListBlobsFlatSegmentResponse.Segment.BlobItems { | ||
|
|
||
| if *currentBlob.Name != containerPath { |
There was a problem hiding this comment.
This needs a bit of Encoding - *currentBlob.Name will be in backend encoding where containerPath is in Standard encoding I think.
|
Let me know when you want me to take another look. |
What is the purpose of this change?
It is now possible to use an azureblob storage without "read" permissions, by getting the metadata from a "list" operation instead. Also, --no-head works now also with a prefix set on the remote.
There are three reasons an option why one would avoid HEAD requests on azureblob, with this now mainly implementing the latter one:
Was the change discussed in an issue or in the forum before?
This fixes #6162 and #7027. The issue with --no-head and a prefix wasn't discussed before AFAIK.
Checklist