Feature request: Use adlfs to access public blobs

[raybellwaves]

Originally posted it at https://stackoverflow.com/questions/63856476/adlfs-create-azureblobfilesystem-with-credential-none but should probably post here. Need rep points to create a adlfs tag and fsspec tags...

I'm going through this:
https://azure.microsoft.com/en-us/services/open-datasets/catalog/goes-16/

and I was curious how to do the equivalent of s3fs.S3FileSystem(anon=True) with adlfs.AzureBlobFileSystem.

It seems credential=None is a good candidate (https://docs.microsoft.com/en-us/python/api/azure-storage-blob/azure.storage.blob.containerclient?view=azure-python) but as far as i'm aware it hasn't been implemented?

>>> import adlfs
>>> fs = adlfs.AzureBlobFileSystem(credential=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ray/local/bin/anaconda3/envs/test_env/lib/python3.8/site-packages/fsspec/spec.py", line 52, in __call__
    obj = super().__call__(*args, **kwargs)
TypeError: __init__() missing 1 required positional argument: 'account_name'

[raybellwaves]

I believe public data in Azure is stored in a slightly different way to AWS.

For example,GOES-16 on az is stored at https://goes.blob.core.windows.net. What's the nomenclature for this in az/adlfs connection_string? what you call the goes part? storage name? blob name?
I see an example here: BlobEndpoint=https://storagesample.blob.core.windows.net;. Once you have that you can access the data in the noaa-goes16 container. e.g. the path to a file would be noaa-goes16/ABI-L2-MCMIPF/2020/001/00/OR_ABI-L2-MCMIPF-M6_G16_s20200010000216_e20200010009536_c20200010010028.nc

For reference if you mount the blob the Storage Explorer you see addition info when copying the file
{"CloudHub.Azure.Storage.Blobs":{"connectionString":"BlobEndpoint=https://goes.blob.core.windows.net;SharedAccessSignature=","containerName":"noaa-goes16","subscription":null,"accountUri":"https://goes.blob.core.windows.net","sourceFolder":"ABI-L2-MCMIPF/2020/001/00/","items":[{"relativePath":"OR_ABI-L2-MCMIPF-M6_G16_s20200010000216_e20200010009536_c20200010010028.nc","snapshot":""}],"sasToken":"","service":"blob"}}

In AWS (and using s3fs) you can grab a file (note files do not overlap as have different frequency) as:

import s3fs

fs = s3fs.S3FileSystem(anon=True)
file = "OR_ABI-L2-MCMIPC-M6_G16_s20200010001184_e20200010003557_c20200010004112.nc"
fs.get("noaa-goes16/ABI-L2-MCMIPC/2020/001/00/{}".format(file), file)

Therefore would probably need extra arg
adlfs.AzureBlobFileSystem(credential=None, storage_name="goes")

That's because other datasets are hosted on different storage
e.g. https://azure.microsoft.com/en-us/services/open-datasets/catalog/hls/ is at https://hlssa.blob.core.windows.net

[lostmygithubaccount]

are there any workarounds to read from public blobs currently? trying to read from Azure open datasets hosted on a public blob

[hayesgb]

I've added anonymous authentication to AzureBlobFileSystem with #106 and #107 so the above goes example works as follows:

storage_options = {'account_name': 'goes'}
fs = AzureBlobFileSystem(**storage_options)
fs.ls("noaa-goes16")

An example Jupyter Notebook on Azure can be found here. The above doesn't work if you just try fs.ls("/").

I believe @raybellwaves is right -- the trick is making sure you get the account_name and container right.

BTW -- The GOES data is stored in xarray, so I didn't try opening the file.

Also successfully accessed the data here as follows:

storage_options={"account_name": "hlssa"}
fs = AzureBlobFileSystem(**storage_options)
fs.ls("hls")
ddf = dd.read_csv("abfs://hls/S2_TilingSystem2-1.txt", storage_options=storage_options)

Both .head() and .tail() work
tag: @lostmygithubaccount

[lostmygithubaccount]

will this be published to pypi soon @hayesgb ?

[hayesgb]

I’m trying to get to releases on the first week of the month. You can install what’s in the master branch directly with pip from github if you need it now.

…

On Sep 24, 2020, at 12:28 PM, Cody ***@***.***> wrote:  will this be published to pypi soon @hayesgb ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.