feat(drift): add --tree for structural version-to-version comparison - #76
Conversation
`drift` compares two documents. This answers a different question: what
structurally changed between two releases of the same artifact.
The distinction is the whole point. Every signal in the npm dogfooding study
read a package's *presentation* — declares a repository, has a description,
looks mature — and every one was blind to a compromised *established*
package, because a hijacked real package presents perfectly. All 84 misses
shared that profile. What distinguishes them is not state but change:
version N looked one way, version N+1 grew a payload.
vajra drift ./pkg-1.0.0 ./pkg-1.0.1 --tree \
--input-format source --lang javascript
Files are matched on path *relative to each root*, so the differently-named
extraction directories of two tarballs align instead of reporting every file
as both added and removed. Comparison is by structural shape, so
reformatting and identifier renaming do not register while an added branch
or a new call does. Reports added/removed/changed/unchanged counts, per-file
node deltas, and a net delta; unchanged files are omitted from the per-file
list but counted in the summary.
Validated on real releases. Two adjacent versions of an npm package flagged
as malicious, against a normal patch of a legitimate one:
legitimate patch flagged release
files changed 2 of 6 (33%) 59 of 61 (97%)
net node delta +153 -391,898
Three orders of magnitude apart. In that case the earlier version was the
obfuscated one and the later was clean, so the direction ran opposite to
what I first assumed — which is itself the point: a patch release that
rewrites the structure of 97% of its files is not a normal patch whichever
way it runs, and the docs say how to read the two columns together. A large
positive delta concentrated in one or two files is the signature of an
injected payload; a large delta spread across nearly every file is a
build-pipeline change.
Limits documented rather than implied: only files the format selector
accepts are compared, so a payload in a shell script or binary is invisible;
parse failures are reported and counted as changed rather than silently
treated as unchanged (deeply nested obfuscated code can exceed the
tree-sitter recursion limit, which happened on one real file); and shape
equality is not semantic equality.
Tests: 8 unit (identical trees, relative-path matching, added/removed/
changed, value-only edits unchanged, node-delta accounting, unparseable file
reported, non-directory rejected, determinism) and 9 CLI integration tests
including that renaming is not a structural change and that an injected
payload is detected in an otherwise stable release.
|
Warning Review limit reached
Next review available in: 5 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
1 participant
Closes #62.
The gap this closes
Every signal in the npm dogfooding study read a package's presentation — declares a repository, has a description, looks mature. All 84 misses at the best operating point shared one profile: established-looking projects with real GitHub URLs, 15+ scripts, and test suites. A hijacked real package presents perfectly, so presentation-based detection is blind to it by construction.
What distinguishes those packages is not their state but their change: version N looked one way, version N+1 grew a payload. Nothing in vajra could express that.
vajra drift ./pkg-1.0.0 ./pkg-1.0.1 --tree \ --input-format source --lang javascript --format json --quietValidated on real releases, with a control
Two adjacent versions of a real npm package flagged as malicious, against a normal patch release of a legitimate one:
cli-table30.6.4→0.6.5)@ejazullah/browser-mcp0.0.57→0.0.58)Three orders of magnitude apart. The control matters — "59 files changed" is meaningless without knowing what a normal patch looks like.
A finding I got backwards at first, worth recording. I assumed 0.0.58 had been obfuscated. Reading the files, the reverse is true: 0.0.57 carries
_0x5082-style identifier mangling and 0.0.58 is clean Apache-licensed Microsoft code. The node count fell because obfuscation inflates AST nodes — consistent with the original study's finding. That doesn't weaken the result: a patch release that rewrites the structure of 97% of its files is not a normal patch whichever direction it runs, and knowing a package just switched its build pipeline is worth an alert either way. The docs now explain how to read the two columns together — a large delta concentrated in one or two files is the signature of an injected payload; a large delta spread across nearly every file is a build-pipeline change.Design
package/, and two extraction directories have different names, so comparing absolute paths would report every file as both added and removed. Tested.unchanged— without that property every release would look like a rewrite and the signal would be worthless.Limits, documented rather than implied
.jsonby default, source under--input-format source. A payload dropped in a.shor a binary is invisible here.changed, never silently treated as unchanged. This is not hypothetical:package/lib/program.jsin the real corpus exceeds the tree-sitter recursion limit on both sides, and lands inerrors.Tests
8 unit tests: identical trees, relative-path matching, added/removed/changed detection, value-only edits report unchanged, node-delta accounting for added and removed files, unparseable file reported and counted changed, non-directory input rejected on both sides, determinism.
9 CLI integration tests: identical releases, renaming is not a structural change, injected payload detected in an otherwise stable release with a positive delta isolated to one file, differently-named roots align, added/removed reported with specified ordering, JSON trees without
--input-format, two directories required, text output states the comparison basis, byte-identical repeat runs.Full workspace suite: 77 test binaries, 0 failures.
cargo fmt --checkclean. 0 clippy diagnostics invajra-cliunder--all-targets --all-features.