Support for MS Fabric OneLake

[mikaelene]

Microsoft recently announced Fabric Online which is based on Azure Data Lake Storage Gen2. It should be quite compatible with ADLS Gen2 APIs and SDKs to be compatible with existing ADLS Gen2 applications. But it looks like it has a new protocol (abfss?).

It would be great to have support for OneLake and maybe some docs on how to upload files etc.

[chriklev]

Is this supported now?

[jkafrouni]

A hacky workaround I've found is:

from adlfs import AzureBlobFileSystem

fs= AzureBlobFileSystem(account_name="onelake", anon=False)
fs.account_host = "onelake.blob.fabric.microsoft.com"
fs.do_connect()

You can then pass it for example to pandas to read abfs:// paths:

df = pd.read_parquet(path, filesystem=fs)

Related to #340

[kevinjqliu]

Quick update on the status of Microsoft OneLake support

Microsoft OneLake is API-compatible with both Azure Blob Storage and Azure Data Lake Storage. This means users can interact with OneLake through the Blob endpoint (onelake.blob.fabric.microsoft.com) and the DFS endpoint (onelake.dfs.fabric.microsoft.com).

Both @.jkafrouni and #486 confirm that this library works with the OneLake Blob endpoint.
However, the OneLake DFS endpoint is not yet supported. This is not an inherent limitation of OneLake; rather, adlfs itself does not support Azure Data Lake Storage APIs (see issue #401).