trio.Path.iterdir wrapping is broken #501
Comments
You're right, good catch. |
Also this will require changing the signature – right now it's an async method that returns a synchronous iterator, and it needs to become a synchronous method that returns an async iterator. I think we can just make the change if we do it soon – you're probably the first person to hit this, and the old behavior is simply wrong. |
It feels like there's no real reason that the Doing some fancy wrapping of the stdlib function feels like it'd be a lot trickier. It would essentially entail something like the following, which was ripped from the stdlib almost verbatim: async def iterdir(self):
func = partial(os.listdir, self._wrapped)
files = await trio.run_sync_in_worker_thread(func)
def gen():
for name in files:
if name in {'.', '..'}:
continue
yield self._wrapped._make_child_relpath(name)
if self._wrapped._closed:
self._wrapped._raise_closed()
return gen() |
Hmm, I assumed that the point of this function was that it didn't pull the whole directory listing into memory at once. Is that wrong? |
No, it just calls def iterdir(self):
"""Iterate over the files in this directory. Does not yield any
result for the special paths '.' and '..'.
"""
if self._closed:
self._raise_closed()
for name in self._accessor.listdir(self):
if name in {'.', '..'}:
# Yielding a path object for these makes little sense
continue
yield self._make_child_relpath(name)
if self._closed:
self._raise_closed() where |
Anyway, so doing it this way would change the semantics slightly as we'd be getting the directory listing when creating the generator object rather than when we later start iterating over it, but (Furthermore, the |
Hmm, tricky. The underlying operating systems do expose incremental reads for large directories. And Python even exposes it via There are some extra complications though:
....So I think maybe I've convinced myself that Proposed TODO list for this bug
|
@njsmith I've submitted a PR for this |
Given
pathlib.Path.iterdir
returns a generator that does IO access on each iteration,trio.Path.iterdir
is currently broken given it currently only generates the generator asynchronously (which I suppose is pointless given there is no need for IO at generator creation)The solution would be to modify
trio.Path.iterdir
to return an async generator, however this means creating a special case given the current implementation is only an async wrapper onpathlib.Path.iterdir
.The text was updated successfully, but these errors were encountered: