preserve order of input files in output without sacrificing parallelism

[jikamens]

There should be a way to tell rg to preserve the order of input files specified on the command line in the output.

E.g., if I'm searching a bunch of files which are in chronological order, I want the output to be in chronological order.

[BurntSushi]

You can do this today, but you have to give up parallelism. If you run rg -j1 PATTERN FILE FILE ... then the output should be deterministic and in the order specified. In general, when recursively searching a directory, the order is whatever the file system serves up, but with -j1 it should at least be deterministic.

[jikamens]

Yeah, so I don't want to give up parallelism, obviously.

GNU parallel let's me be parallel and preserve the order of output with --keep. I'm proposing that rg should provide similar functionality. Otherwise rg -j# isn't a drop-in replacement for grepping a bunch of files with GNU parallel, which I would really like it to be. ;-)

[BurntSushi]

I get what you're saying. Here are my thoughts on the matter:

1. If deterministic ordering + parallelism were possible to do without incurring additional costs, rg would do that by default. (Because I'd really like deterministic ordering too.) Perhaps I am wrong and this is possible (or perhaps, possible with immeasurable costs), and if so, we should do that.
2. If not (1), then hiding this behind another flag seems like bad UX, but maybe that's the best we can do.
3. Requiring single threaded searches for when you absolutely need deterministic output doesn't seem unreasonable to me. For example, rg can sometimes beat or be competitive with ag using a single thread even when ag runs with multiple threads.

All of these things combined together means I'm not particularly motivated to work on this, but I grant it's something I'd be willing to explore given unbounded resources.

[jikamens]

You can implement ordering + parallelism with no additional cost; all rg has to do is read output from the threads in the order that they are launched. This means, of course, that some of the threads will block waiting to be read because their output buffers will fill up, but the performance of such an implementation would be no worse than -j1, and in most cases it would be significantly better.

You can improve the performance simply by making the output buffers larger so that they block less.

"You just have to put up with your search running eight times slower on your eight-core machine," is not really a reasonable answer, especially since I don't have to put up with it, since as I pointed out, there are other tools that will allow me to search and preserve ordering in the output.

I frankly get the impression that you are rationalizing this idea being not worthwhile because the internal implementation of rg is not conducive to it. That's putting the cart before the horse. If it's too hard to do this right now in rg, fine, that's a reasonable answer. But to then try to hand-wave away the value of the idea because it's hard to do is a step too far.

[BurntSushi]

You can implement ordering + parallelism with no additional cost; all rg has to do is read output from the threads in the order that they are launched. This means, of course, that some of the threads will block waiting to be read because their output buffers will fill up

Unless I'm missing something, it sounds like the additional cost there is both memory and time.

but the performance of such an implementation would be no worse than -j1, and in most cases it would be significantly better.

But it sounds like it will be slower than -j8 on an eight core system, which doesn't quite satisfy "no additional cost." With that said, it's plausible that there is no measurable cost, but it's hard to know that without trying an experiment.

Comparing your suggestion to -j1 is interesting, but that's not the standard we need to live up to make this the default behavior. Simply being better than -j1 is good enough to put it behind a flag, but it's not good enough to make it default. I know you requested putting it behind a flag, but I'd like to see it be default behavior, and this is the frame of mind I have when inquiring about costs.

"You just have to put up with your search running eight times slower on your eight-core machine," is not really a reasonable answer, especially since I don't have to put up with it, since as I pointed out, there are other tools that will allow me to search and preserve ordering in the output.

Of course it's a reasonable answer. This is about time preferences and priorities. Two reasonable people can disagree on how important this particular issue is. I happen to think it's not that important. You clearly do. That's OK.

If GNU parallel lets you achieve this already, then it sounds like you can use ripgrep with GNU parallel to get everything you want except for the convenience of using one tool instead of two.

I frankly get the impression that you are rationalizing this idea being not worthwhile because the internal implementation of rg is not conducive to it.

I'm sorry I gave you that impression because that wasn't my intent. I think we should try to avoid reading into others' motivations and give the benefit of the doubt that I'm saying what I mean. Without an assumption of good faith here, it's hard to have a productive conversation. (To be frank, this comment by you is putting me on the defensive and making it hard for me to respond.)

To respond though, I don't think rg's current implementation really has anything to do with this. The multithreaded component of rg is pretty small (look at src/main.rs), and it wouldn't take much effort to rewrite it. In fact, I plan on rewriting at least a portion of it soon, because I'd like to make directory iteration itself parallelized. I suspect this may make deterministic output even harder. The best I can do is say that I'll give this issue more thought when I parallelize the directory iterator.

[jikamens]

Point taken. Thanks for the thoughtful response.

[jikamens]

If GNU parallel lets you achieve this already, then it sounds like you can use ripgrep with GNU parallel to get everything you want except for the convenience of using one tool instead of two.

Unfortunately, I frequently get "Out of memory (os error 12)" from ripgrep when I try to use "rg -j1" with GNU parallel.

[BurntSushi]

@jikamens Well... that's bad. :P If you'd be willing to file a bug, I'd be happy to look into it. I'm not a user of GNU parallel, so I think I'd need at least the full command you're running. If you can reproduce the issue on data we can both access, that's even better, but I understand if that's not possible.

[jikamens]

I've just told you all that I'm able to tell you, so you can file the bug yourself if you want. ;-)

Seriously, the reason I haven't filed a bug is because the data I'm searching is very large and proprietary and so are the search strings I'm using, so all I'd be able to say in the bug is, "Yeah, I'm getting Out of memory (os error 12) but I can't tell you want I'm doing to provoke it." If I have time and am able to construct a test case that I can share, I'll open an issue.

[ryanberckmans]

@BurntSushi on ripgrep 0.4.0,

       --sort-files
              Sort results by file path.  Note that this currently disables 
              all parallelism and runs search in a single thread.

It would be great if --sort-files with parallelism was the default. Sorting results by file path is a big benefit for me. Let me know if there's a better place for this feedback :).

Thanks for all your hard work! ripgrep is a joy to use. We have type coupling between microservices so I am often searching for FooRequestType and rely on file path order to parse results by service.

[BurntSushi]

@ryanberckmans There's no debate that --sort-files with parallelism would be better than --sort-files without parallelism from an end user's perspective. I mean, it's not like I just arbitrarily chose to disable parallelism for the hell of it.

Glad you're enjoying ripgrep. :-)

[BatmanAoD]

Since the documentation states that search time is guaranteed to be linear, perhaps we'd get more consistency (without guaranteeing full determinism of output) by ordering the files to search from largest to smallest by default? This of course would require completing the directory traversal before even starting a search, but I'm not sure how big a burden that is, either in terms of programming effort to make the modification or runtime memory usage to store and traverse the file list. (I'm guessing that --sort-files does not have this limitation, since you don't need to inspect child directories before starting to search through files in a parent directory.)

Obviously, to guarantee consistency, there's no way to escape the fact that sometimes the output will "hang" while waiting for the next thread to complete its search.

[BurntSushi]

Storing the entire tree in memory isn't a path I'm excited about pursuing. It's not only a huge code change, but it will introduce additional latency and a memory requirement that is proportional to the size of the tree.