Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prevent HEAD call for Azure Blob #7027

Open
nicottin opened this issue May 24, 2023 · 1 comment
Open

Prevent HEAD call for Azure Blob #7027

nicottin opened this issue May 24, 2023 · 1 comment
Labels
enhancement help wanted Remote: Azure Blob

Comments

@nicottin
Copy link

Hi,
In our project, for security reason, we need to upload files to an Azure Blob storage with a user having only write authorisation (no read).
In this situation, the upload works but at the end, it is always failing because I guess there is a check with those HEAD calls but since our user has no read authorisation, it finishes with 403.
Would there be a possibility to add an option to avoid this check and HEAD call?
I have seen something similar with S3 but nothing for Azure. (--s3-no-head )

Thanks !

  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.
@edc-w
Copy link

edc-w commented Jul 3, 2023

Hi there @nicottin
Thanks for your post. This sounds like a useful enterprise feature that you are after.
We have no plans to add this currently but would be happy to help you out with this and get it implemented via consultation.
If you would like us to implement this for you then please send us an email to sales@rclone.com and we'll see what we can do for you.

moqmar added a commit to moqmar/rclone that referenced this issue Apr 30, 2024
@moqmar
azureblob: allow working with only list+create/write permissions (fix…
d1904f0 
… no_head_object/add no_read_for_metadata)

There are three reasons an option why one would avoid HEAD requests on azureblob, with this now implementing the latter one:

- no_head_object: Don't do a HEAD *before* writing to (or reading from) an object, to reduce the number of transactions and thus increase performance. **This is quite complex and thus not implemented here, see s3.Object.Update()!**
- no_head: Don't do a HEAD *after* writing to an object, for the same reason but at the cost of "increasing the chance for undetected upload failures". This already exists in azureblob, a small issue was fixed here.
- no_read_for_metadata: Instead of doing a HEAD to get metadata, use a list operation, to avoid requiring READ permissions for immutable/append-only applications (this seems to be quite azureblob-specific) - one could also use both no_head_object and no_head together to achieve this, but then uploads wouldn't be verified.

This hence fixes rclone#6162 and rclone#7027, as well as an as of now unreported issue where the --no-head option wouldn't work at all with a prefix set on the remote.
moqmar added a commit to moqmar/rclone that referenced this issue Apr 30, 2024
@moqmar
azureblob: allow working with only list+create/write permissions (fix…
a42930b 
… no_head_object/add no_read_for_metadata)

There are three reasons an option why one would avoid HEAD requests on azureblob, with this now implementing the latter one:

- no_head_object: Don't do a HEAD *before* writing to (or reading from) an object, to reduce the number of transactions and thus increase performance. **This is quite complex and thus not implemented here, see s3.Object.Update()!**
- no_head: Don't do a HEAD *after* writing to an object, for the same reason but at the cost of "increasing the chance for undetected upload failures". This already exists in azureblob, a small issue was fixed here.
- no_read_for_metadata: Instead of doing a HEAD to get metadata, use a list operation, to avoid requiring READ permissions for immutable/append-only applications (this seems to be quite azureblob-specific) - one could also use both no_head_object and no_head together to achieve this, but then uploads wouldn't be verified.

Also fixes an as of now unreported issue where the --no-head option wouldn't work at all with a prefix set on the remote.

Fixes rclone#6162 and rclone#7027
moqmar added a commit to moqmar/rclone that referenced this issue Apr 30, 2024
@moqmar
azureblob: allow working with only list+create/write permissions (fix…
839ca0e 
… no_head_object/add no_read_for_metadata)

There are three reasons an option why one would avoid HEAD requests on azureblob, with this now mainly implementing the latter one:

- no_head_object: Don't do a HEAD *before* writing to (or reading from) an object, to reduce the number of transactions and thus increase performance. **This already exists in azureblob, a small issue was fixed here where using a prefix would still require a HEAD request to the prefix itself.**
- no_head: Don't do a HEAD *after* writing to an object, for the same reason but at the cost of "increasing the chance for undetected upload failures".  **This is still not implemented for azureblob yet and quite complex, see s3.Object.Update()!**
- no_read_for_metadata: Instead of doing a HEAD to get metadata, use a list operation, to avoid requiring READ permissions for immutable/append-only applications (this seems to be quite azureblob-specific) - one could also use both no_head_object and no_head together to achieve this, but then uploads wouldn't be verified.

Fixes rclone#6162 and rclone#7027
@moqmar moqmar mentioned this issue Apr 30, 2024
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement help wanted Remote: Azure Blob
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Animosity022 @nicottin @edc-w and others