glep 2.0.0

ripgrep parity and a parallel search pipeline: most of rg's flags with rg's semantics, checked byte for byte against rg on both the single-thread and the worker-pool search paths, and a first crates.io release since 0.2.3.

2.0.0 follows 0.3.1 directly; there was no 0.4.0 release.

Upgrading from 0.3.1:

• Existing indexes rebuild once, on the first query (postings format v2 to v5).
• Run in a subdirectory of an indexed tree, glep now uses the nearest .glep/ above it and scopes the search to the current directory; 0.3.1 built a second index in the subdirectory.
• -l, -c and --json are last-wins output modes, as in rg; 0.3.1 rejected -l or -c with --json, and -c with -l, as a usage error.
• Binary detection follows rg's reader (a NUL in the first 64 KiB, where 0.3.1 looked at 8 KB), and UTF-16 files with a byte-order mark are transcoded and searched instead of skipped.
• .rgignore files apply, and --no-ignore-vcs also turns off .git/info/exclude and the global gitignore, so results can change in trees that rely on them.

Performance

• Worker-pool search: one reusable searcher per worker thread, an in-order printer thread with a bounded window, whole-file slice searches and coalesced writes, so output stays byte-identical to rg --sort path at unordered parallel throughput. The freshness diff runs sharded in parallel over an unsorted sweep, and on macOS --files answers from a dir-mtime listing cache. Every flag below runs on this pipeline; flags whose bytes are not the file's own (--pre, -z, --binary, --no-mmap) read through the searcher's buffer inside the worker.
• Files the sweep flags hidden are no longer re-checked against every ignore file on the way to them unless something can still show them (--hidden, a -g, -t or --ignore-file whitelist, or a hidden operand): the flag already holds every ignore file's ! rescue. A default --files listing or full-scan search over a tree with many dotfiles in many directories paid about 20 us of ignore-file syscalls per hidden file (on a tree with 2,000 .gitignore files, --files took 54 ms instead of 11 ms).
• On macOS the search worker pool defaults to the performance-core count (hw.perflevel0.physicalcpu) instead of one worker per logical core when -j is absent or 0. A full scan is mostly kernel open, read and page-fault work whose lock contention grows faster than the thread count, so efficiency cores add contention rather than throughput: on an M3 Pro (5 performance plus 6 efficiency cores) a kernel-tree full scan now spends about a quarter of the system time and finishes in roughly half the wall time (glep q 2.44 s to 1.35 s, glep -c ab 2.70 s to 1.15 s, glep static 1.24 s to 0.78 s, against rg's 2.36 s, 2.40 s and 2.31 s on the same busy machine). GLEP_THREADS=N overrides the default and an explicit -j N still wins; elsewhere (Linux, Windows, Intel Macs) the default stays one worker per core. The rayon pool behind sweeps and listings keeps one thread per core, where a --files listing runs about 4% faster than at the performance-core count.
• Search workers read each file of up to 1 MiB whole into one reused buffer and search it as a single slice, instead of memory-mapping every file. A file under 64 KiB now costs an open, two reads and a close; a map also cost an fstat, an mmap, a fault per page and an munmap, and mmap and munmap serialize the workers on the process's VM map lock. Bigger files are still mapped: copying them costs more than faulting them in, and reading files up to 8 MiB made a narrowed kernel search whose candidates include the multi-megabyte amdgpu register headers about a third slower (--ttl 60 PCI_EXP_LNKCTL2_TLS 25 ms to 35 ms). Each worker also searches with its own clone of the matcher, which shares the compiled program but owns its scratch space, so workers no longer queue on the regex engine's shared cache pool, and a file's path is built in a reused buffer. Output is unchanged. On the kernel tree, one interleaved hyperfine invocation per row, mean: glep -c '.*' 2.05 s to 1.80 s, -c '\w+' 1.52 s to 1.38 s, q 1.45 s to 1.36 s, static 892 ms to 848 ms (rg: 2.48 s, 2.61 s, 2.84 s, 2.68 s); at -j 4, -c '.*' takes 1.90 s against rg's 2.14 s.
• A --files listing scoped to path operands (or run from a subdirectory) lists straight from a live sweep of its operands and never opens, builds or writes the index. The scoped sweep the index path ran already produced exactly the files it listed, so loading and diffing the manifest (about 11 ms on the kernel tree, whatever the scope) bought nothing, and on an unindexed tree a scoped listing no longer builds the whole index first. On macOS the sweep starts at each operand: the levels above it are read by name (their .gitignore, .ignore and .rgignore, then the next level's stored name and type) instead of listing every directory on the way down one after another, several operands walk in parallel, and a listing reads directories with readdir (names and types only) on a pool sized like the search pool. A scoped search's freshness sweep starts at its operands the same way. Output is unchanged. On the kernel tree, one interleaved hyperfine invocation per row, mean (min) in ms for the previous build, this one and rg: --files '**' drivers/net 22.4 (21.2), 7.9 (7.2), 10.1 (9.1); drivers 48.0 (46.4), 26.7 (24.8), 32.1 (30.8); fs/ext4 include/linux 23.0 (22.0), 6.0 (5.5), 7.3 (6.3); scripts/dtc/include-prefixes, which holds only symlinks, 16.6 (14.0), 3.2 (2.7), 4.9 (3.8).
• The first query on an unindexed tree holding more than 64 MiB no longer waits for the index build. It answers from the sweep that measured the tree, at full-scan cost, the way --no-index answers from its walk, and once its output is flushed it starts the build in a detached child at background priority (utility QoS on macOS, nice 10 on other Unix systems). The child's stdio is the null device, so a pipe or $(...) reading the query sees the end of its output as soon as the query exits, and it runs in a process group of its own, so Ctrl-C at the terminal leaves it alone. Queries that arrive during the build answer from their own sweep; the first one after it opens the index. A smaller tree still builds inline, now from that same sweep instead of sweeping twice, and no query runs the freshness sweep right after building the index itself. GLEP_SYNC_BUILD=1 keeps the inline build and GLEP_BG_BUILD_MIN_BYTES moves the threshold. Windows keeps the inline build, since a child there inherits the query's stdout pipe and its reader would wait for the build. On the kernel tree, over fresh clones of it in one hyperfine invocation, the first query went from 26.4 s to 2.6 s for PCI_EXP_LNKCTL2_TLS and from 25.5 s to 2.6 s for static, against rg's 1.5 s and 1.8 s there. That first query is a full scan, which on this machine ran between a third of rg's time and 1.7 times it as kernel contention came and went (1.1 to 1.9 s while rg took 2.8 to 3.3 s). The build took 22 to 24 s in the background, and the query after it took 150 ms against rg's 1.57 s.
• A root-wide --files listing on macOS takes about half as long when the dir-mtime listing cache answers it. The pruned walk no longer copies every cached leaf directory and then reassembles, compares and frees a whole new tree on each run: it records cached leaves by reference, notes whether any directory it lists live differs from its cached node, and assembles and saves a tree only then. -g, --iglob, -t/-T, --ignore-file, --max-depth and the hidden rule apply inside the listing, before any path is copied, so --files -g '*.h' copies and sorts 25k paths of the kernel tree instead of 86k, and the listing and the loaded cache are left to the process exit instead of being freed one allocation at a time. A cache built after a full listing also no longer trusts a directory or ignore-file mtime from after its sweep started: a file created, or a .gitignore edited, while the cache was being built stayed hidden from every later listing until its directory changed again. Output is unchanged. On the kernel tree, one hyperfine invocation per row, mean (min) in ms for the previous build, this one and rg: --files -g '*.h' 68.7 (64.2), 33.1 (29.2), 75.9 (67.0); --files 64.7 (62.7), 31.7 (30.4), 72.1 (70.1); --files '**/*.c' 60.8 (59.7), 30.8 (29.4), 70.3 (68.4); and with a new file created in a leaf directory before every run, so that every run rewrites the cache, --files -g '*.h' 78.0 (72.7), 54.3 (51.0), 81.1 (78.6).
• A search whose pattern no trigram can narrow (a pattern shorter than three bytes, .*, \w+, [0-9]{2}, or any pattern under -v, --passthru, -c --include-zero, a non-UTF-8 -E or --pre) chooses its plan before the index opens, and over the whole tree on macOS no longer opens the index at all: it searches the file set the dir-mtime listing cache lists, filtered as the index's would be, instead of loading the manifest and running the full freshness sweep only to get every file back (about 170 ms of work before the first file was searched on the kernel tree, now about 45 ms). Every listed file is searched and binary files are left to the searcher's binary detection, as in rg, so on this path --stats and --json count them as searched, as rg does. The manifest and its sweep epoch are left alone, so the next narrowed query still sweeps; with --ttl such a search keeps the index path, which answers inside its window without a sweep, and on a cache miss the index answers as before. On a tree with no index yet such a search, on every platform, walks the tree as --no-index does and builds no index: the first query an index can narrow builds it. Output is unchanged. On the kernel tree, in round-robin runs of the previous build, this one and rg (8 rounds, mean ms): -c ab 1238, 1093, 2498; -c '.*' 1890, 1778, 2363; -c '[0-9]{2}' 1667, 1402, 2340; `--files-witho...

glep v0.3.1

glep 0.3.1 updates the package links to the new documentation site. There are no code changes since 0.3.0.

What changed

• Homepage, documentation, changelog and repository links on PyPI now point to https://velofy.co/glep/ and https://github.com/velofy/glep.
• The README was rewritten, and the old GitHub Pages site now forwards to the new documentation.
• Adds a CHANGELOG.md file.

Install

pip install glep==0.3.1

Wheels are on PyPI for macOS (arm64, x86_64), Linux (x86_64, aarch64, musl x86_64) and Windows (amd64), and are attached here. crates.io currently serves 0.2.3, which lacks --hidden, --no-ignore and the JSON summary event. To build this version with cargo: cargo install --git https://github.com/velofy/glep --tag v0.3.1.

Documentation

https://velofy.co/glep/