large file uploads fail with SAS URI connected container

[eschultink]

Storage Explorer Version: Both 1.1 and 1.2 (tried both)
Platform/OS Version: macOS and Windows (we've had people try on both platforms, although I'm personally on macOS)
Architecture:
Version: 1.2.0

Platform: macOS
Architecture: x64
Build Number: 20180604.2
Commit: f8373d5a
Support Id: 87e505ce-9d60-4576-5c5e-dfb4dccd49a9

Copyright © 2018 Microsoft. All rights reserved.
Regression From: Not sure it ever worked. We've tried on various machines and various versions.

Steps to Reproduce:

1. create a SAS URI for a bucket, with wdl perms
2. connect it by right clicking on "Storage Accounts" --> "Connect ..", choosing SAS URI method, pasting in the URI in there. Otherwise all defaults.
3. Upload with defaults. Small file works; Large file fails.

{
  "name": "StorageError",
  "message": "Forbidden",
  "stack": "StorageError: Forbidden\n    at Function.StorageServiceClient._normalizeError (/Applications/Microsoft Azure Storage Explorer.app/Contents/Resources/app/node_modules/azure-storage/lib/common/services/storageserviceclient.js:1191:23)\n    at Object.StorageServiceClient._processResponse (/Applications/Microsoft Azure Storage Explorer.app/Contents/Resources/app/node_modules/azure-storage/lib/common/services/storageserviceclient.js:738:50)\n    at Request.processResponseCallback [as _callback] (/Applications/Microsoft Azure Storage Explorer.app/Contents/Resources/app/node_modules/azure-storage/lib/common/services/storageserviceclient.js:311:37)\n    at Request.self.callback (/Applications/Microsoft Azure Storage Explorer.app/Contents/Resources/app/node_modules/request/request.js:185:22)\n    at emitTwo (events.js:126:13)\n    at Request.emit (events.js:214:7)\n    at Request.<anonymous> (/Applications/Microsoft Azure Storage Explorer.app/Contents/Resources/app/node_modules/request/request.js:1157:10)\n    at emitOne (events.js:116:13)\n    at Request.emit (events.js:211:7)\n    at IncomingMessage.<anonymous> (/Applications/Microsoft Azure Storage Explorer.app/Contents/Resources/app/node_modules/request/request.js:1079:12)",
  "code": "Forbidden",
  "statusCode": 403,
  "requestId": "f09047e2-501e-0013-070d-04d352000000"
}

Expected Experience:

It should work for both small and large files.

Actual Experience:

It works only for small files.

[jinglouMSFT]

What's the approx size of the larger files to repro this? Also there was a period of time yesterday Azure Storage was down.

[eschultink]

So with a SAS token that also has READ permissions, it works. Given this, I would guess that it uses a different upload mechanism for large files rather than small files; that upload mechanism tries to READ somehow, which fails?

[eschultink]

70MB+

[eschultink]

it is current, happens now.

[craxal]

Yeah, this is a known issue. Here's what happens.

For particularly large files (like your 70+MB), we want to know if a blob of that name already exists before trying to upload the file. Otherwise you could be waiting a few minutes just for the platform to tell you the blob already exists before dropping the uploaded file!

So, before uploading anything, we just ask the platform if the blob exists using the Get Blob Properties operation. Unfortunately, this means you need read permissions. In most cases, this isn't an issue. But in your case, your SAS needs read permissions. Otherwise larger uploads will fail.

See #140 for more info.

[eschultink]

OK, very confusing to have it work differently depending on the file-size - especially when it provides no useful feedback about the error.

The original token did have LIST permission as #140 indicates is required - just not WRITE.

Use-case is that I want customers to be able to securely upload data files. Hence we send them a URI that allows them to write only - so they can send us the files, but if that URI was ever compromised, it would not allow someone to read those files.

Also strange that the "Create SAS" wizard allows WRITE ONLY tokens to be created, if those have no utility.

[craxal]

In your case, the key is read permissions, not list.

We'll need to spend some time looking into this. We've gotten reports of this several times, so we'll want to look at our options. It looks like the platform supports just overwriting blobs if they already exist, but that may be undesirable in some cases. I'm sure, for example, you don't want users overwriting other users' blobs when they happen to upload something with the same name.

[eschultink]

well, we're not going to hand different customers SAS tokens for the same container. We create a container per customer. So that risk is minimal. And in any case, I think it's our responsibility to consider that rather than yours.

But regardless:

1. the behavior of this shouldn't vary based on file-size. If READ is needed to WRITE, it should ALWAYS be required to write. Otherwise I create a SAS token, test it, it works fine for me - so I send to customer - and then it fails for them. We look stupid.
2. if you allow a write-only token to be created, it should work for writing - the potential overwrite case is the user's problem. If you don't want to allow that, your UX shouldn't allow me to check the "WRITE" box without also having checked the "READ" box. Mysterious failures on the other end isn't correct.
3. there should be some permission short of READ that allows for reading of blob meta-data, but not the blob itself. I thought that's what the LIST permission did, but I guess not?

[craxal]

List permissions let you ask the container what blobs it has. Read permissions let you get specific data from the blob, including content and metadata.

Don't worry; we're with you on this one! We're aware of the problem, and we intend to fix it without breaking other user needs. We've been chatting, and we think we have a simple solution. Theoretically, we should be able to just query the container for blobs using the target blob as a prefix. If there are no matching blobs returned, then we don't have to worry about overwriting. This should release the requirement for read permissions.