Skip to content

callbacks: replace kwargs positional argument with arbitrary kwargs in branched #1496

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 15, 2024
Merged

callbacks: replace kwargs positional argument with arbitrary kwargs in branched #1496

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions fsspec/callbacks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ def __exit__(self, *exc_args):
def close(self):
"""Close callback."""

def branched(self, path_1, path_2, kwargs):
def branched(self, path_1, path_2, **kwargs):
"""
Return callback for child transfers

Expand All @@ -58,8 +58,8 @@ def branched(self, path_1, path_2, kwargs):
Child's source path
path_2: str
Child's destination path
kwargs: dict
arguments passed to child method, e.g., put_file.
**kwargs:
Arbitrary keyword arguments
Comment on lines +61 to +62
Copy link
Contributor Author

@skshetry skshetry Jan 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before, this could be used to influence kwargs passed to put_file/get_file. I am not sure what the use case for that was, except of adding callback kwarg.

So this is deviating with branch in that regard.

Copy link
Member

@martindurant martindurant Jan 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you vote to remove and add back later if there is a use case?

Copy link
Contributor Author

@skshetry skshetry Jan 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a use case for mutating kwargs in branched(). So, I'd just go with **kwargs (i.e. this PR).

I don't have any strong opinion against kwargs=None as it is similar but less ergonomic. But it does provide a way of mutating kwargs (but I don't see a use case for it).

We could also drop the third argument entirely, but that is not very flexible for future changes (I know I started with that, but 🤷🏽).


Returns
-------
Expand All @@ -77,7 +77,7 @@ def branch_coro(self, fn):

@wraps(fn)
async def func(path1, path2: str, **kwargs):
with self.branched(path1, path2, kwargs) as child:
with self.branched(path1, path2, **kwargs) as child:
return await fn(path1, path2, callback=child, **kwargs)

return func
Expand Down
4 changes: 2 additions & 2 deletions fsspec/spec.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -967,7 +967,7 @@ def get(

callback.set_size(len(lpaths))
for lpath, rpath in callback.wrap(zip(lpaths, rpaths)):
with callback.branched(rpath, lpath, kwargs) as child:
with callback.branched(rpath, lpath) as child:
self.get_file(rpath, lpath, callback=child, **kwargs)

def put_file(self, lpath, rpath, callback=_DEFAULT_CALLBACK, **kwargs):
Expand Down Expand Up @@ -1053,7 +1053,7 @@ def put(

callback.set_size(len(rpaths))
for lpath, rpath in callback.wrap(zip(lpaths, rpaths)):
with callback.branched(lpath, rpath, kwargs) as child:
with callback.branched(lpath, rpath) as child:
self.put_file(lpath, rpath, callback=child, **kwargs)

def head(self, path, size=1024):
Expand Down
6 changes: 2 additions & 4 deletions fsspec/tests/test_callbacks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,13 +37,11 @@ def test_callbacks_as_context_manager(mocker):

def test_callbacks_branched():
callback = Callback()
kwargs = {"key": "value"}

branch = callback.branched("path_1", "path_2", kwargs)
branch = callback.branched("path_1", "path_2")

assert branch is not callback
assert isinstance(branch, Callback)
assert kwargs == {"key": "value"}


@pytest.mark.asyncio
Expand All @@ -55,7 +53,7 @@ async def test_callbacks_branch_coro(mocker):

assert await wrapped_fn("path_1", "path_2", key="value") == 10

spy.assert_called_once_with("path_1", "path_2", {"key": "value"})
spy.assert_called_once_with("path_1", "path_2", key="value")
async_fn.assert_called_once_with(
"path_1", "path_2", callback=spy.spy_return, key="value"
)
Expand Down