Get-AzureStorageBlobContent to variable (Feature Request)

[jboeshart]

Cmdlet(s)

Get-AzureStorageBlobContent

PowerShell Version

Instructions: to get PowerShell version, type $PSVersionTable and look for the value associated with PSVersion
all

Module Version

all

OS Version

Instructions: to get OS version, type $PSversionTable and look for value associated with BuildVersion
all

Description

Currently, you can only use this command to download blob content to a file. It would be beneficial if functionality could be added to save blob content to a variable instead of to a local file, for cases where you may not have access to the local file system or may only want the data to be in memory and not to disk.

Debug Output

Instructions: to get Debug Output, set $DebugPreference="Continue" and then execute the cmdlet or script causing the issue

Script/Steps for Reproduction

N/A, new feature request

[cormacpayne]

@blueww would you mind taking a look at this feature request?

[blueww]

@jboeshart
Would you please share more details for your scenario, such as how you will use the data in memory, which cmdelts you will use to handle the variable contains the content.
It will help us better understand your requirement.
Thanks!

[jboeshart]

Sorry for the delay, missed your comment here @blueww. In my case I wanted to get JSON content straight out of an NSG Flow Log that is sitting on Azure Storage, parse through the JSON and then upload it to OMS. Presently, I have to send that blob content to a local file, read it back in and then do the parsing. If I could get that blob content straight to a variable it would save the step of having to write to a file and read it back in, and could help in cases where I may not have access to or may not want to write to a local filesystem (Azure Functions, Azure Automation, etc.). Ideally I'd like to be able to do something like this:

$blobContent = Get-AzureStorageBlobContent -Container $ContainerName -Context $storageContext -Blob $blob.Name

[blueww]

@jboeshart
We have opened a task to track the issue and will triage it according to our bandwidth.

Please note, as the blob content might be very big, we will consider to transfer it to a stream (e.g. memory stream) instead of put the text to a variable

[jboeshart]

Thanks @blueww. I'm not familiar with using a stream, how does that impact the usability of the data that is in the blob?

[blueww]

You can new an System.IO.MemoryStream object in PowerShell, and we storage cmdlet will download blob content to the memory stream, and you can read from the memory steam.
You can see some sample to use memory stream in PowerShell in :
https://social.technet.microsoft.com/Forums/windowsserver/en-US/7d838d53-bfd5-4706-b0df-c4bee2d00412/using-memorystream-and-textreader?forum=winserverpowershell

[mariotoffia]

I'm facing the same scenario where I use the blob store with lots of JSON documents that I read and take actions on. It being implemented in azure functions and would like to have the ability to read in memory. If I may - it would be nice to use e.g. a switch like -AsString that will pass on the contents of the blob as a string and thus can pipe it to convertfrom-json directly and thus do e.g. like this
$blobs | Get-AzureStorageBlobContent -AsString | ConvertFrom-Json ...

I understand that blobs may be really BIG but in my case they are rather small and should fit in memory just fine :)

Cheers,
Mario

[blueww]

@mariotoffia
As you know, PowerShell is a common tool, so we need to consider many facts when convert blob content to string, like big blob, string encoding ...

I will add your comments to the track item for this issue.
Before we get a solution on this, can you use:

Get-AzureStorageBlobContent -Destination $filename
get-content $filename | ConvertFrom-Json 

[mariotoffia]

@blueww Yes, that exactly what I've done - a custom commandlet for this :)

I understand that you've targeting a broad audience - but if you use a flag that indicates a In-Memory (maybe with a constraint of maximum 4096/8192 bytes or so) I don't see the problem. Or will this drive support issues due to people don't read the help text?

It actually would ease the load on azure-functions on Microsoft server farms due to no IO on disc and (possibly) network (if network attached storage) for storing temporary files :)

Cheers,
Mario

My custom hack (not in memory though):

$file = New-TemporaryFile 
$blob = Get-AzureStorageBlobContent -Container $name -Blob $f -Context $AuthCtx.StorageContext -Destination $file -Force

$contents = Get-Content $file -Raw -ErrorAction:SilentlyContinue
Remove-Item $file -Force -ErrorAction:SilentlyContinue

$contents

[jboeshart]

@blueww How does reading the blob content straight to memory differ from writing it to a file then reading it back in if we're talking about a large blob file? Wouldn't the same concerns around memory apply here, just that you're pushing the problem to Get-Content instead of Get-AzureStorageBlobContent?

I need to play around with writing to a memory stream to see if that works. It does add some code smell though when talking pure PowerShell, as it's not as intuitive a process as leveraging a cmdlet parameter.