test: migrate golden specs to Backspin.compare - #107
Merged
Conversation
0.13.0 adds Backspin.compare for differential testing. Regenerates the Appraisal gemfiles and lockfiles along with the root lockfile.
Each of these ran the same record name through Backspin.run twice - once for rspec, once for plur - so the recorded snapshot acted as a hand-rolled differential test. Backspin.compare does that directly: one call, one chdir, one filter, nothing persisted to disk. Drops the workarounds that existed only for the two-call pattern: - the "args" => ["[SOMETHING_COMMAND]"] placeholder in every filter. compare never compares argv, so normalizing it is dead code. - expect(result.verified?).to be(true). A mismatch raises VerificationError with the diff before the expectation is reached. - the three single-failure examples, which differed only by record name and were otherwise byte-identical. Collapsed to one, keeping the union of the color, backtrace, and description assertions. Deletes the five snapshot YAMLs those specs recorded into. The remaining fixtures/backspin records are ordinary single-command snapshot tests and are unaffected.
This pair recorded in one example and verified in a different one, so it passed only under RSpec's defined order and broke under --order random. One compare call makes that failure mode structurally impossible. The comparison is still pending - plur's output does not match raw minitest output yet - and compare's raise on mismatch registers as pending-as-expected rather than an error.
Appraisal2 emits hashrocket syntax; standard:fix restores the 1.9 hash syntax the committed gemfiles already used.
Follow-up cleanup to the Backspin.compare migration. No behavior change. - Delete the `fixture_path` aliases, which just forwarded to the suite-wide `project_fixture` helper. - Inline the single-use command builder methods. Collapsing the duplicate examples left each one with exactly one call site, and `reference:`/`actual:` already name what they are. - Collapse the two-layer normalizers. `make_summary_line_consistent`, `normalize_timing`, and `normalize_minitest_output` each had a single caller in the snapshot filter directly below them. - Drop the dead plur-banner regexes from the pending-output filter. They scrubbed stdout, but plur writes that banner to stderr - and the worker regex required a plural "workers" that never matched the single-worker runs these specs produce. - Inline the minitest seed and drop the `seed:` keyword; it existed to make a recording deterministic and nothing is recorded now. - Note why each filter blanks stderr, so the next reader doesn't have to rediscover that plur's version/worker banner has no rspec counterpart. 348 -> 294 lines.
single_failure_spec.rb was a hand-rolled Backspin.compare: it ran the same two commands against the same fixture, kept its own copy of the timing normalizer, and asserted stdout_lines == rspec_stdout_lines. The comparison is strictly weaker than compare's - no exit status, no stderr. compare's result carries the real captured output, so the absolute assertions that comparison alone can't make - exit status, the failure count, plur's stderr banner - hang off result.actual in the same example. No extra subprocess pair to make them. Removes 4 subprocess spawns per suite run and the third copy of make_summary_line_consistent. --color=never runs stay covered by colorized_output_spec.rb and tty_output_spec.rb.
It compared plur's full minitest output against raw grouped minitest output and could never pass. Two of the differences are deliberate decisions pinned by passing specs in this same file - the RSpec-style duration line, and consolidating progress dots onto the first line - and the dot/puts interleaving position is not stable anyway, since plur's worker command passes no seed. The one real defect it pointed at is that minitest mode drops a test's own stdout. That is tracked in plur-internal (docs/reviews/2026-07-26-minitest-stdout-dropped.md); once fixed it wants a direct assertion, not a full-output diff.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrates the rspec-vs-plur golden specs to
Backspin.compare(new in backspin 0.13.0) and deletes the workarounds that only existed because there was no differential-testing API.57 insertions, 448 deletions. Specs alone: -272/+35.
The pattern being removed
Each of these specs called
Backspin.runtwice with the same recordname:— once for rspec, once for plur — so a recorded snapshot acted as a hand-rolled differential test:Backspin.comparedoes this directly: one call, onechdir, one filter, nothing persisted to disk.Three workarounds fall out with it:
"args" => ["[SOMETHING_COMMAND]"]placeholder in every filter.comparenever compares argv, so normalizing it was dead code.expect(result.verified?).to be(true). A mismatch raisesBackspin::VerificationErrorwith the diff before the expectation is reached, so the assertion could never fail — only fire after the raise already had.single_failure_golden_spec.rbthat differed only by record name and were otherwise byte-identical.Six snapshot YAMLs are deleted. The remaining six in
fixtures/backspin/are ordinary single-command snapshot tests usingBackspin.runcorrectly and are untouched.Order-dependence fixed
minitest_integration_spec.rbrecorded in one example and verified in a different one, so it passed only under RSpec's defined order and broke under--order random. Onecomparecall makes that failure mode structurally impossible.single_failure_spec.rbremovedIt was a hand-rolled
Backspin.compare: same two commands, same fixture, its own copy of the timing normalizer, assertingstdout_lines == rspec_stdout_lines. Strictly weaker thancompare, which also checks exit status and stderr.Its absolute assertions — exit status, failure count, plur's stderr banner — moved onto
result.actualin the compare-based example, so they cost no extra subprocess pair.--color=nevercoverage remains incolorized_output_spec.rbandtty_output_spec.rb.A pending spec dropped rather than migrated
The
plur -n1vs raw-grouped-minitest comparison could never pass: the RSpec-style duration line and consolidated progress dots are deliberate decisions pinned by passing specs in that same file, and the dot/putsinterleaving position isn't stable because plur's worker command passes no seed. The real defect it pointed at — minitest dropping test stdout — is fixed separately in #106.Verification
Full suite green in both defined and random order. Each migrated spec was confirmed to still fail when it should: pointing plur at
--color=neveragainst rspec's--force-colorraisesVerificationErrorwith a real diff in all three golden specs.Worth noting the migration made the suite faster, not slower.
Backspin.runshells out in verify mode too, so the old code was already re-running rspec on every test run and comparing it against a snapshot of itself. Collapsing the duplicate examples removed 4 subprocess spawns per run.Also bumps backspin to
~> 0.13.0and regenerates the Appraisal gemfiles and lockfiles. Appraisal2 emits hashrocket syntax, so there is a follow-upstandard:fixcommit restoring the 1.9 hash syntax those files already used.