Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

performance bug: ripgrep does poorly when given lots directory arguments #44

Closed
BurntSushi opened this issue Sep 24, 2016 · 2 comments
Closed
Labels
bug A bug.

Comments

@BurntSushi
Copy link
Owner

ripgrep is doing some non-trivial work for every file searched in positional parameters, and there's really no reason it should.

@BurntSushi BurntSushi added the bug A bug. label Sep 24, 2016
@BurntSushi
Copy link
Owner Author

In the github.com/rust-lang/rust repo, there is a pretty big time difference:

$ time rg E0401 | wc -l
7

real    0m0.072s
user    0m0.393s
sys     0m0.087s
$ time rg E0401 src/lib* | wc -l
6

real    0m0.202s
user    0m1.623s
sys     0m0.007s

If -u is passed to the second command, then the time difference goes away.

This means ripgrep is spending a lot of time reinitializing ignore state. There are some obvious things we shouldn't be redoing, like the globs and type filters from the command line, but those happen even with the -u flag, so they aren't the problem. The likely problem is re-traversing parent directories for gitignore files. Indeed, if --no-ignore-parent is given, then the performance difference goes away.

This means we need to keep our gitignores from parent directories around and try to reuse them.

@BurntSushi BurntSushi changed the title performance bug: ripgrep does poorly when given lots of positional arguments performance bug: ripgrep does poorly when given lots directory arguments Oct 29, 2016
BurntSushi added a commit that referenced this issue Oct 30, 2016
This PR introduces a new sub-crate, `ignore`, which primarily provides a
fast recursive directory iterator that respects ignore files like
gitignore and other configurable filtering rules based on globs or even
file types.

This results in a substantial source of complexity moved out of ripgrep's
core and into a reusable component that others can now (hopefully)
benefit from.

While much of the ignore code carried over from ripgrep's core, a
substantial portion of it was rewritten with the following goals in
mind:

1. Reuse matchers built from gitignore files across directory iteration.
2. Design the matcher data structure to be amenable for parallelizing
   directory iteration. (Indeed, writing the parallel iterator is the
   next step.)

Fixes #9, #44, #45
@BurntSushi
Copy link
Owner Author

Fixed in #202.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug A bug.
Projects
None yet
Development

No branches or pull requests

1 participant