Skip to content

Short-circuit 'easy' rglob/glob patterns#276

Merged
pjbull merged 1 commit into
drivendataorg:276-live-testsfrom
Gilthans:rglob-special-case
Nov 2, 2022
Merged

Short-circuit 'easy' rglob/glob patterns#276
pjbull merged 1 commit into
drivendataorg:276-live-testsfrom
Gilthans:rglob-special-case

Conversation

@Gilthans

Copy link
Copy Markdown
Contributor

Resolves #274

@Gilthans

Copy link
Copy Markdown
Contributor Author

@pjbull friendly ping :)

@pjbull

pjbull commented Oct 20, 2022

Copy link
Copy Markdown
Member

Thanks @Gilthans, while I think this fix looks directionally reasonable, but I'd like to do a bit more investigation on why performance is bad in #274.

I assumed that it was because there was a network call somewhere in the glob path, but we actually only call the list dir once here:

def _glob(self, selector):
root = _CloudPathSelectable(
PurePosixPath(self._no_prefix_no_drive),
{
PurePosixPath(c._no_prefix_no_drive): is_dir
for c, is_dir in self.client._list_dir(self, recursive=True)
},
is_dir=True,
exists=True,
)

If you get a chance, could you do some profiling and report in #274? I'm hoping there may be a better more general fix than the short circuit if this isn't networking related (or maybe it is and I just missed where the call is in my scan of the code).

@Gilthans

Copy link
Copy Markdown
Contributor Author

I assumed that it was because there was a network call somewhere in the glob path, but we actually only call the list dir once here:

I actually came to the same conclusion before opening the issue. I got that far into the code and couldn't figure out why it was slow either.
I'd love to do this profiling (one of my hobbies actually :) ), but in the meantime, maybe we could push forward with this? It seems to me like a net-positive regardless

@pjbull pjbull changed the base branch from master to 276-live-tests November 2, 2022 01:05
@pjbull pjbull merged commit bd5445e into drivendataorg:276-live-tests Nov 2, 2022
pjbull added a commit that referenced this pull request Nov 2, 2022
* Short-circuit 'easy' rglob/glob patterns (#276)

* Add change description

Co-authored-by: Daniel Oriyan <gilthans@gmail.com>
@codecov

codecov Bot commented Apr 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.4%. Comparing base (b7f6010) to head (9e56b09).

Additional details and impacted files
@@               Coverage Diff                @@
##           276-live-tests    #276     +/-   ##
================================================
- Coverage            94.8%   94.4%   -0.4%     
================================================
  Files                  20      20             
  Lines                1314    1323      +9     
================================================
+ Hits                 1246    1250      +4     
- Misses                 68      73      +5     
Files with missing lines Coverage Δ
cloudpathlib/cloudpath.py 93.7% <100.0%> (+0.1%) ⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

glob and rglob can be slow for large directories or lots of files

2 participants