support searching compressed files using in-process decompression

[danbst]

I'd like to use ripgrep for grepping log files, because it's faster then grep. But my logs are gzipped, and if I zcat | rg them I'll loose log filenames in output.

Also, would be great if bzip2 and xz decompressors will be supported too with automatic archive type detection.

[BurntSushi]

I'm torn on this one for both pragmatic and philosophical reasons.

Philosophically, this is making ripgrep do a lot more than just "open a file and search it." I don't actually believe that ripgrep should do everything, even though it already does a lot. Nevertheless, it's hard to argue with the usefulness of this feature.

Practically speaking, there are two primary issues as I see it:

1. If we wanted to implement this right now, we'd need to bring in C libraries. While there's no hard requirement against doing this (to my knowledge, as long as the three main platforms are supported), I would like to actively discourage it in the interest of keeping ripgrep pure Rust.
2. This requires some kind of UX design. How does ripgrep detect compressed files? When it finds them, does it always search them or does a user explicitly need to enable it? How are decompression options, if they exist, controlled? What is "automatic archive type detection"?

I think that if we were to do this, it is at least blocked on some initial implementation of #1, since supporting additional text encodings has a lot of overlap with decompressing files before searching in terms of making the core search routines work with it.

With that said, I don't personally see myself working on this any time soon.

[alejandro5042]

Brainstorming here...

Instead of rg constantly playing catching up on every type of compressed file option, an alternative is to have a plugin system whereby the user can decide what rg does when it encounters a file. For example, rg can look up the extension in a configuration file (or the glob in a .gitattributes-type of file), execute the preprocessor step, runs the search on the results, then deletes it. Alternatively, it can simply stream the output of the command in memory and skip the file writes. Seems like quite a bit of work though....

Anyway, this could also be used to search Word, binary, XML, or other complicated formats; if for example you could turn the complicated format into plain-text format that rg can happily parse.

[BurntSushi]

As the maintainer, I'm not particularly interested in the plugin path, sorry.

[danbst]

Ok, I see. Also, I don't think that ripgrep -z will perform much better then zgrep, because decompression takes time and levels the ripgrep speed (unless Rust can magically do faster decompression).

Now, none of ag, rg and zgrep can replace each other - zgrep is faster then ag on compressed files and beats rg, ag can do searches in gzip, zip, lzma and xz (with autodetection), which none of zgrep and rg do, and rg is fastest when no compression is used and your regex feats declared expressivity.

[BurntSushi]

I don't think that ripgrep -z will perform much better then zgrep

That might be true on single files (it really depends on what proportion of time is spent in decompression), but ripgrep can certainly get wins with parallelism.

Now, none of ag, rg and zgrep can replace each other

Can we not use this as motivation for adding new features please? I ask this because it will always be true. For example, if you need a POSIX compliant search tool, then neither ripgrep nor ag will suffice. If you need backreferences or lookaround, then ripgrep won't work but grep -P and and ag will. I don't think that will ever change. ripgrep will never be everything to everyone and that's OK.

With that said, it's certainly reasonable to expect some convergence of features. For example, I specifically built ripgrep so that folks could use it for both the "search a large repo of code" and "search very large files" use cases.

I think my initial comment on this issue still stands: this feature is a possibility, but has a few philosophical and practical problems with it.

[wavexx]

I would expect at least basic unix stream compressors to be supported (.gz/.bz2/.xz). All editors decompress (and often recompress) those on the fly.

Debian ships share/doc/* files in compressed form when doing so results in space savings. I couldn't do a quick search there, and searching through docs is something I do often. It makes perfect sense to have documentation compressed.

Source is sometimes compressed too. Emacs compresses .el files by default on install as well. Ironically, using emacs ripgrep package I cannot search into an installed emacs lisp files ;)

I expect a performance hit when searching through compressed files, so I don't think that's a problem. My main concern is that I could miss a match because one of the files has been compressed. For text data files, this happens frequently, especially in repositories that need to float over the network.

[BurntSushi]

@wavexx I don't think there's any question that this is a desirable feature. Thank you for sharing your use cases though, they're helpful.

[moorereason]

I'm troubleshooting a cups server issue (for 3 days now), and I wish I could use a feature like this. Some files within the cups installation are gzipped while others are not, and they're scattered all of the place. So far I've been doing find . -type f -name "*.gz" | xargs zgrep needle, but I'm not finding what I expect to see and feel like I may be missing something.

Regarding the issues Andrew brought up earlier: 1) I'd rather not have ripgrep linked to C libs 👎 and 2) for the UX design, I'd just offer a -z option and have ripgrep work in "zip-only" mode. That would be good enough for my use.

I'm a nobody, but my vote is to freeze this issue until pure-Rust compression libs are available and then re-evaluate this idea at that point.

[rik]

On macOS, zgrep is the 2.5.1 BSD version. BSD grep is noticeably slower than GNU grep. I haven't found a way to easily install the GNU version so it would be nice to have a fast and convenient to grep directories of compressed log files.

[BurntSushi]

@rik I think you can install the GNU tools through brew? (I'm not a Mac user though, so don't take my word for it. I think you wind up needing to use ggrep for GNU grep, for example.)

[rik]

Sorry, I should have mentioned that I've looked into Homebrew/homebrew-dupes. Yes you can do that but that only installs grep, not zgrep.

[Xenofex]

I use ag and sift for my everyday search, not rg just because it doesn't support gzip. This UX issue matters a lot when I do a lot of log search as a sysadmin. Generally log files are larger than code files. Now I have 3.6GB gzipped log files to search frequently, where the raw speed of a search tool really shines. As of my code, I don't really matter if it is ag, sift or rg, because every of them gives me the result fast enough.

As a search utility focusing on speed, I think log file search should be its target use case, where your effort devoted really helps.

Thanks!