Fix missing kwargs injection on upload blob call

[nffdiogosilva]

Hello there.

Currently, when uploading a blob, there's no way of injecting other arguments besides the data, overwrite, and metadata.

An example would be when I want to compress data and set the content encoding of the data to br. For that I need to pass a ContentSettings object into the upload_blob method kwargs:

i.e:

from azure.storage.blob import ContentSettings
import brotli
import fsspec

CONN_STRING = "<CONN_STRING>"
CONTAINER = "kwargs-injection"

abfs = fsspec.filesystem(
    "abfs",
    connection_string=CONN_STRING,
)

if __name__ == "__main__":
    content_settings = ContentSettings(
        content_type="application/json",
        content_encoding="br",
    )
    abfs.mkdir(CONTAINER, exist_ok=True)
    abfs.write_bytes(
        path=f"{CONTAINER}/compressed.br",
        value=brotli.compress(b'{"a": "hello world"}', quality=1),
        overwrite=True,
        **{"content_settings": content_settings},
    )

What I'm expecting to see in the azure storage containers web interface, when consulting the blob properties:

What I see instead (no content encoding defined):

script requirements:
$ pip install fsspec adlfs brotli

[TomAugspurger]

xref fsspec/filesystem_spec#916. But agreed that forwarding arguments makes sense, even if that were to be standardized.

We should document this somewhere though. Maybe a short note in index.md?

[nffdiogosilva]

@TomAugspurger that makes sense to standardised but in the meantime can we merge this PR so it can unblock my current use case?

Regarding the note, not sure what you are suggesting. I've added this below the usage section in the index.md:

Note: When uploading a blob (with write_bytes or write_text) you can injects kwargs directly into upload_blob method:

>>> from azure.storage.blob import ContentSettings
>>> fs = adlfs.AzureBlobFileSystem(account_name="ai4edataeuwest")
>>> fs.write_bytes(path="path", value=data, overwrite=True, **{"content_settings": ContentSettings(content_type="application/json", content_encoding="br")})

Was it something like this that you wanted?

[TomAugspurger]

that makes sense to standardised but in the meantime can we merge this PR so it can unblock my current use case?

Yep. I think we'll want this PR regardless of what happens upstream, to support other keyword arguments.

It seems that there's a linting check failing. I can't actually view the Azure Pipelines output (cc @hayesgb). Maybe you can try running the linters locally to see what's failing? This looks good otherwise.

[nffdiogosilva]

For reference: https://levelup.gitconnected.com/fix-runtimeerror-poetry-isort-5db7c67b60ff

[nffdiogosilva]

Hello @TomAugspurger. It seems that the tests were failing due to the isort version that was being used in the pre commit hooks. (you can understand the error better here: https://levelup.gitconnected.com/fix-runtimeerror-poetry-isort-5db7c67b60ff). Fix it now, and the checks are all passed.

Can we merge it now, please?

Thank you.

[TomAugspurger]

Thanks!

[nffdiogosilva]

@TomAugspurger, wasn't a new version built with this fix? If not, can you build one please? Anything I didn't do that was suppose to do? Let me know if I can help. Thank you.

[TomAugspurger]

@hayesgb has handled the last few releases. @hayesgb are you able to make a new release sometime, or would you need me to?

[eirki]

Just a friendly notification: I would be super grateful if someone could build a new version with this fix.