Intermittent: SetAzureContext needs to be called twice in order to retrieve storage key

[chrislill]

Steps to repro:

1. Set valid values for the parameters in the code snippet below
2. Run the code snippet
   Result: Error message occurs
3. Run the SetAzureContext and dirlist (azureBlobLS) commands again
   Result: The directory list is retrieved

Error message:

> Error: 
Fetching Storage Key..
Fetching Storage Key..
Error in AzureSMR::azureBlobLS(sc, directory = "", recursive = FALSE,  : 
  Error: No storageKey provided: Use storageKey argument or set in AzureContext

Code snippet:

sc <- AzureSMR::createAzureContext();
AzureSMR::setAzureContext(sc,
                          tenantID = tenantID,
                          clientID =clientid,
                          authKey=authKey,
                          subscriptionID=subID, 
                          resourceGroup= rG, 
                          storageKey = storageKey, 
                          storageAccount = storageAccount);
AzureSMR::azureAuthenticate(sc);
AzureSMR::azureCheckToken(sc);
dirlist <- AzureSMR::azureBlobLS(sc, directory= "", recursive = FALSE,container = "input");

[kjohnson4]

This had consistently failed for me, whether I provide a storageKey, or used azureAuthenticate(sc) to get a key. Valid values are placed in the "" for each param. I followed your steps, and added a setAzureContext() step, ran that, then ran the azureListStorageBlobs(), and it then worked.

> sc <- createAzureContext(tenantID = "", clientID = "", authKey = "")
> azureAuthenticate(sc)
[1] "Authentication Suceeded : Key Obtained"
> azureListStorageBlobs(sc, resourceGroup=" ", storageAccount="", container=" ")
Fetching Storage Key..	
Fetching Storage Key..
Error in azureListStorageBlobs(sc, resourceGroup = "",  : 
  Error: No storageKey provided: Use storageKey argument or set in AzureContext

OR

> sc <- createAzureContext(tenantID = "", clientID = "", authKey = "")
> azureListStorageBlobs(sc, resourceGroup=" ", storageAccount="", storageKey=”” container=" ")
Fetching Storage Key..
Fetching Storage Key..
Error in azureListStorageBlobs(sc, resourceGroup = "",  : 
  Error: No storageKey provided: Use storageKey argument or set in AzureContext

[Alanwe]

Thanks for raising the issue.

I'm struggling to repo it. Is this also Intermittent for you?

Can you check the storageKey value in Azure context before and after azureListStorageBlobs using str(sc). I'm also interested in any functions you are calling before running azureListStorageBlobs that maybe setting the key to NULL.

IF you are running createAzureContext followed directly by azureListStorageBlobs, then it maybe something environmental. Try running

azureSAGetKey(sc, resourceGroup="YYY",storageAccount="XXXX", verbose=TRUE)

Thanks Alan

[kjohnson4]

When I run createAzureContext(), and then run azureAuthenticate(sc), it says a Key is obtained, but running str(sc), there is no entry for storageKey; so I guess its NULL?
I can see a storageKey if I run the setAzureContext() with the storageKey explicitly stated as a param. str(sc) verifies it is there.

So, creating a context, authenticating, and attempting azureListStorageBlobs() does not work. Creating a context, and specifying a storageKey in the azureListStorageBlobs() does not work. This happens every time. It isn't until I run setAzureContext() with a storageKey param that this will work.

I'm not running anything else in-between these commands, so I don't think anything could be 'nullifying' the storageKey.

[yueguoguo]

@chrislill Chris I have tired your code and it worked for me. Can you try azureSAGetKey to see whether you can get a valid storage key? Guess problem was from there.
@Alanwe Function of azureSAGetKey does not have an error handle after POST - the request may fail but the process will continue and that is why a NULL is returned!
@kjohnson4 In createAzureContext() there are no option to set service-specific parameters like storageKey. You need to specify these parameters either in setAzureContext or the actual functions, i.e., azureBlobLS or aureListStorageBlobs.

[kjohnson4]

@yueguoguo See the second example in my first post, where I tried providing a key in azureListStorageBlob, and it still failed. Just tried it again, and it fails with same message.

[Alanwe]

@kjohnson4 I was wondering if your Storage Account was created in classic mode. At the moment it only supports the newer API. I think Chris is right that the error handling in in azureSAGetKey needs tightening.

[kjohnson4]

I'll have to check if the SA was created in classic, but all our resources are provisioned via Resource manager now, and resources that were in classic have been moved.
I can successfully perform the following operations; would I be able to do these if this was classic storage?:

azureBlobLS(sc, directory="mydir", storageAccount="mystorage", container="mycontainer")
rawBlob <- azureGetBlob(sc, blob = "somefileblob", type = "raw")
txtBlob <- azureGetBlob(sc, blob = "somefileblob", type = "text")

[Alanwe]

@kjohnson4 No if it were classic I don't think they would work either.
One point of note relating to @yueguoguo comment you should never have to actually provide a storageKey as the API should fetch the key automatically.

This example may help, it would be good to know if it works for you.

sc <- createAzureContext(tenantID=MYTID,clientID=MYCID,authKey=MYKEY)
azureCreateResourceGroup(sc,"myResourceGroup","northeurope") # Create Resource Group
azureCreateStorageAccount(sc,storageAccount="myazuresmrteststorage",resourceGroup = "myResourceGroup")
# Wait for the storage account to provision
azureListSA(sc,resourceGroup = "myResourceGroup")

azureCreateStorageContainer(sc,"myteststorage")
library(RCurl)
flightfile <- getURL("https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/244708/Home_Office_Air_Travel_Data_2011.csv")
azurePutBlob(sc,contents=flightfile,blob = "flightinfo/flightfile.csv")

azureBlobCD(sc,"/flightinfo")
azureBlobLS(sc)
azureDeleteBlob(sc,blob="flightfile.csv")
azureDeleteResourceGroup(sc,resourceGroup = "myResourceGroup")

But I would love to get to the bottom of your issue.

Thanks Alan

[kjohnson4]

I skipped the first 3 lines (createResourceGroup and CreateStorageAccount) because I dont have permissions to create RGs, and due to AzureSMR has only been setup for authorization on the one SA that I've been using currently ( with Read&Write access).
However, all the other commands, after your "Wait for the storage account to provision" line worked just fine! I was able to verify the blob using AzureStorageExplorer.
I'll continue commenting on other issue opened for my continued issues trying to use azurePutBlob.

[yueguoguo]

@kjohnson4 Yea your second use was correct. The cause might be invalid account, key, or even typos. As azureSAGetKey function does not handle error, we might not have been aware of errors when there are. :)

To create resource group, you need to add access at subscription level. It can be done exactly the same as what you did with access control under a resource group.

1. Go to Azure portal.
2. Find subscription on the left pane. If there is no, click More services.
3. Add access of your Active Directory Application to the subscription (all the same as resource group).

Hope helps!

[kjohnson4]

Thanks for steps to setup permissions at subscription level, however our IT team does not intend to provide this service that level of access; my use-case only requires blob storage read/write. The creation of resources falls outside this process and something I don't intend to do with this package.

[Alanwe]

Thanks for the feedback, The Azure Systems Management API is indeed different from the Azure Storage API and shouldn't need any AD access. I will take your use case and look to ensure the storage functions can work effectively without having to access the Systems Management API.

[andrie]

@chrislill Can you please test again using the latest version (v0.2.4)?

All of the functions now perform error checking code when Azure responds, so if an error happens at least AzureSMR should tell you what the problem is.