rg-opendal, a ripgrep-style search CLI over OpenDAL objects

[TennyZhuang]

Background

ripgrep ships its core matching/searching/printing logic as public crates on crates.io: grep-searcher, grep-regex, grep-matcher, grep-printer, ignore. Those crates expose stable APIs for embedding ripgrep's regex engine and output behavior in other tools without forking ripgrep itself.

rg-opendal uses those crates, with OpenDAL as the I/O layer, to bring ripgrep-compatible search to S3-compatible object stores. The CLI takes the form rg-opendal <pattern> s3://bucket/prefix/ and aims to mirror native rg's flag set and output format on object-store inputs.

Current status

Early-stage feasibility study. The crate is a single repository with [lib] + [[bin]] structure plus a separate benchmarks/seam-a/ microbenchmark crate. Verification is shell-harness-based: each feature family has a script that diffs the binary's output against native rg on a fixture hosted in a local MinIO instance.

Two reader strategies are wired:

• A full-buffer reader (default), which is the fastest path on local MinIO in our measurements.
• A streaming reader (--streaming) that bridges OpenDAL's BufferStream to sync std::io::Read via Handle::block_on. Local benchmark on 2.7–525 MB files measured the per-chunk bridge cost at ≤1% of total search time; the dominant cost is S3 I/O.

Implemented scope

The following ripgrep flag families have feature-parity-attested implementations against native rg's output on a MinIO fixture (the per-family status table is in MATRIX.md):

• Literal and regex search; case-insensitive (-i).
• Glob and type filtering (-g, -t, -T); -t coverage is currently unit-test-level rather than full output-parity.
• JSON output, context lines, stats (--json, -A/-B/-C, --stats).
• Color output (--color=auto|always|never).
• Count and files-with-matches (-c, -l).
• Word and line regexp (-w, -x).
• Max-count (-m).
• Invert match (-v).
• Printer flags: line-number, column, heading (-N, --column, --heading).
• Path-prefix control with auto-detection (-I, -H).
• NUL path terminator (-0) and NUL-separated records (-z).
• Force-text mode for binary inputs (-a) — flag-acceptance smoke only at this point.
• Fixed-strings (-F) — spot-checked, dedicated harness pending.
• Streaming reader (--streaming), with output-parity-attested against the default reader.

One small convention from working with OpenDAL: each harness probes op.info().full_capability().<flag> and skips a test rather than failing it when the underlying service doesn't support a capability. That keeps the service-vs-feature axes clean during development.

Known limitations

• All current measurements are against local MinIO, not real S3. Real-S3 latency hasn't been characterized.
• CI is not yet configured; verification is run locally.
• Walker is serial-only (no parallel/streaming walker yet).
• The -a/--text family is flag-accepted but not yet output-parity attested against native rg.
• The -F/--fixed-strings family has been spot-checked but does not yet have a dedicated harness.
• The repository is not yet published to crates.io.

Questions for the community

• Is the integration shape (independent crate consuming grep-* from crates.io, with OpenDAL as the I/O layer) reasonable from the OpenDAL maintainers' perspective?
• Has anyone in the OpenDAL ecosystem combined grep-* (or another sync Read-consuming searcher) with BufferStream before? If so, are there pitfalls or conventions on the streaming-bridge pattern we should know about?
• Are there OpenDAL-side conventions for ecosystem tools (badge text, matrix-of-services compatibility, capability-gating language) that we should adopt to keep the project consistent with the rest of the ecosystem?

Repo: https://github.com/TennyZhuang/rg-opendal

Thanks for OpenDAL — happy to take any feedback or pointers.

[github-actions[bot]]

Mailing list thread: https://lists.apache.org/thread/6so6n0jc1s932b9bp349f8nm3mrrkw7x