az storage account generate-sas creating invalid token (Expiry format incorrect)

[nofunatall]

Expiry portion of the SAS token ends up like:
se=2019-12-12T03%3A21%3A25Z

Where it should be like:
se=2019-12-12T03:21:25Z

I have no idea where that format is valid but for most practical uses it seems like complete gibberish (e.g if passing token to azcopy)

Furthermore az cli should generate sas tokens in identical format to the Azure Portal for consistency sake.
Azure Portal generated SAS:
?sv=2019-02-02&ss=bfqt&srt=sco&sp=rwdlacup&se=2119-11-12T10:31:28Z&st=2019-11-12T02:31:28Z&spr=https&sig=REDACTED

Azure CLI generated SAS:
se=2019-12-12T03%3A21%3A25Z&sp=rwdlacup&sv=2018-03-28&ss=bqtf&srt=sco&sig=REDACTED

Command Name
az storage account generate-sas

• az storage account generate-sas --account-name {} --services {} --permissions {} --resource-type {} --expiry {} --output {}

Environment Summary

Linux-5.3.7-050307-generic-x86_64-with-debian-buster-sid
Python 3.6.5
Shell: bash

azure-cli 2.0.76

Extensions:
interactive 0.4.3

[fengzhou-msft]

@Juliehzl please take a look.

[Juliehzl]

@nofunatall Could you tell me if the generated SAS token works for you?

[nofunatall]

Well when using it with azcopy it does not.

I have not tested using a simple curl command.

[yonzhan]

@nofunatall any update for this issue?

[nofunatall]

Been away for awhile but have just tested again with wget and acopy and the SAS tokens generated from az cli don't work with either of them.

#!/bin/bash
EXPIRE=$(date -u -d "1 day" '+%Y-%m-%dT%H:%MZ')
START=$(date -u -d "-1 day '+%Y-%m-%dT%H:%MZ'")
ACCOUNT="some_account"
CONTAINER="some_container"
BLOB="some_blob"
SAS=$(az storage account generate-sas --account-name "$ACCOUNT" --start "$START" --expiry "$EXPIRE" --https-only --permissions cdlruwap --resource-types sco --services bfqt)

echo "Token: $SAS
Account: $ACCOUNT
Container: $CONTAINER
Blob: $BLOB"

echo "Try AZCOPY"
azcopy copy "https://{$ACCOUNT}.blob.core.windows.net/${CONTAINER}/${BLOB}?${SAS}" "$HOME/blobs/test.jpg"

echo "Try WGET"
wget "https://${ACCOUNT}.blob.core.windows.net/${CONTAINER}/${BLOB}?${SAS}" -k -O "test.jpg"

Response from AZCOPY:

failed to perform copy command due to error: cannot use directory as source without --recursive or a trailing wildcard (/*)

Response from WGET:

HTTP request sent, awaiting response... 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
2020-01-07 17:18:53 ERROR 403: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature..

[nofunatall]

I can get WGET working by running sed on the SAS token to convert %3A to :

So it seems like this issue might be with azcopy not az cli
Related azcopy issue:
Azure/azure-storage-azcopy#122

SAS=$(az storage account generate-sas --account-name "$ACCOUNT" --account-key "$KEY" --start "$START" --expiry "$EXPIRE" --https-only --permissions acdlpruw --resource-types sco --services bfqt | sed 's/%3A/:/g;s/\"//g')
wget "https://${ACCOUNT}.blob.core.windows.net/${CONTAINER}/${BLOB}?${SAS}" -k -O "test.jpg"

HTTP request sent, awaiting response... 200 OK
Length: 485190 (474K) [application/octet-stream]
Saving to: ‘test.jpg’

test.jpg 100%[=======================================================================================================================================================================================================================================>] 473.82K 2.26MB/s in 0.2s

2020-01-09 10:31:36 (2.26 MB/s) - ‘test.jpg’ saved [485190/485190]

Converted links in 0 files in 0 seconds.

[Juliehzl]

@nofunatall Thanks a lot for your feedback. If we have time, CLI team can consider adding time details when user only specify the date for sas token generation to satisfy azcopy requirement.

[nofunatall]

This is a snippet of a working example for AzCopy

EXPIRE=$(date -u -d "3 months" '+%Y-%m-%dT%H:%M:%SZ')
START=$(date -u -d "-1 day" '+%Y-%m-%dT%H:%M:%SZ')
SAS=$(az storage account generate-sas --account-name $ACCOUNT --account-key $KEY --start $START --expiry $EXPIRE --https-only --permissions acdlpruw --resource-types sco --services bfqt | sed 's/%3A/:/g;s/\"//g')

[Juliehzl]

I see. Thanks 😊

[Masahigo]

@nofunatall Thanks! Getting AzCopy working from Linux/Docker container is very difficult atm.

[yonzhan]

add to S170

[Juliehzl]

quoting sas token is a operation in python SDK. I see that there is the same quote issue in Storage Explorer. @zezha-msft Could we considering making quoted sas token work in AzCopy?

[zezha-msft]

Hi @Juliehzl, the diagnosis is probably not correct, since SAS generated from Storage Explorer works perfectly fine. Ex:

https://zemaintest.blob.core.windows.net/bar?sv=2019-02-02&st=2020-05-27T09%3A27%3A01Z&se=2020-05-28T09%3A27%3A01Z&sr=c&sp=rl&sig=REDACTED

The SAS should be URL encoded, and it should work with AzCopy. Have you been able to repro this issue on your end?

[Juliehzl]

Hi @Juliehzl Zunli Hu FTE, the diagnosis is probably not correct, since SAS generated from Storage Explorer works perfectly fine. Ex:

https://zemaintest.blob.core.windows.net/bar?sv=2019-02-02&st=2020-05-27T09%3A27%3A01Z&se=2020-05-28T09%3A27%3A01Z&sr=c&sp=rl&sig=REDACTED

The SAS should be URL encoded, and it should work with AzCopy. Have you been able to repro this issue on your end?

I have tried with Azcopy 10.3.4 and the sas token generated with azure cli 2.6.0. it works to me. @nofunatall can you have a try again? and if there is any problem, could you provide log file to help figure out root cause?

azcopy copy "https://zuhlrs.blob.core.windows.net/test1/test0.txt?se=2020-05-30T00%3A00%3A00Z&sp=rwu&spr=https&sv=2018-03-28&ss=b&srt=sco&sig=READACTED" a.txt

[zezha-msft]

A wild hunch here: is it possible that % signs need to be escaped in a script?

[johannespetereit]

cThis happens for me in ubuntu (WSL) in conjunction with az storage blob download.
I don't even need to switch to azcopy for something to break. Need to replace "%3a" with ":"
Pwsh Snippet:

$tokenExpiry = ((get-Date).AddMinutes(1).ToUniversalTime().ToString("s") + "Z")
$key = az storage account keys list -g $rg -n $stacc --query "[0].value" -otsv
# here the workaround is required
$sas = (az storage blob generate-sas --account-key $key --account-name $stacc -c $container -n $blob --permissions r --expiry $tokenExpiry -otsv).replace("%3A", ":")
az storage blob download -c $container -n $blob -f $filename --account-name $stacc --sas-token $sas | out-null

Seems to be wildly inconsistent that I cannot reuse output between the same commadn group (az storage). I'm a bit taken back that this issue is open for 4 years o.O