Support For Microsoft OneLake dfs.fabric.microsoft.com

[keshavksingh]

Microsoft Fabric OneLake have slightly different endpoint f"https://{account_name}.dfs.fabric.microsoft.com" compared ADLS Gen2 f"https://{account_name}.dfs.core.windows.net" and hence adlfs does not support OneLake. It will be helpful to add the support.

[kyleknap]

Per the OneLake documentation, it can be accessed using the existing Azure Storage SDKs and because adlfs uses the SDK under the hood, that property should hold.

I was able to construct an adlfs filesystem leveraging the account_host parameter to point it to the OneLake endpoint:

from adlfs import AzureBlobFileSystem
from azure.identity.aio import DefaultAzureCredential

fs = AzureBlobFileSystem(
    account_host="onelake.dfs.fabric.microsoft.com",
    account_name="onelake",
    credential=DefaultAzureCredential(),
)
with fs.open("<workspace>/<item>.<itemtype>/<path>/<fileName>") as f:
    print(f.read())

I don't have a OneLake account setup to confirm that this succeeds end-to-end, but looking at the HTTP request sent, it appeared to match the documented URL.

@keshavksingh could you confirm this works for you?

[keshavksingh]

It does work

[kyleknap]

@keshavksingh Thanks for confirming! Closing out this issue then. If more people hit issues trying to figure out how to use OneLake with adlfs, we should consider updating the adlfs documentation.

[bolkedebruin]

This does not work for all operations, like iterdir, in my testing.

[kevinjqliu]

@bolkedebruin just curious, were you using the onelake.blob endpoint or the onelake.dfs endpoint?

See this comment about the difference #416 (comment)

[bolkedebruin]

I was using the onelake.dfs endpoint, which is used in the ui of fabric as "abfss://" . Rewriting that url to use blob the connection works. As mentioned in the other issue the onelake protocol is different, and only slightly overlapping.