Skip to content
meganemuraPublic

About

Run what you suspect in a Rails app on base and head, and keep the spec that proves it.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

holdfast

holdfast runs what you suspect about a Rails app instead of leaving it read but not run. You write each suspicion (a bug, a race between two actions, an effect on another area) as a few lines of YAML. holdfast check builds the world with the app's own factories, runs the action on base and head, and answers confirmed with a failing spec to keep, refuted with what it observed, or could_not_run with the exact error. holdfast run checks a change against its base, and hunts with random sequences of the app's entry points when nothing changed.

It is made for a coding agent that reads code and has to say what it found. The skill in skills/holdfast/ is the loop an agent follows.

Quick start in your Rails app

You need Ruby 3.4 or newer, git, and an app whose specs run with RSpec and FactoryBot against a test database on PostgreSQL or SQLite. Run each step from the app's root.

1. Add the gem

group :test do
  gem "holdfast", require: false
end

Then run bundle install. holdfast depends on hegeltest, which the hunt uses.

2. Read, and write what you suspect

The examples below run against the clinic app in test/fixture/clinic: appointments, a job that marks missed appointments as no-shows and charges a fee, and a desk check-in. Reading the job, two things look wrong. It also sweeps tentative appointments, which were never confirmed, and it loads its rows before it locks each one, so a check-in in between may be overwritten. Write both down:

- says: a passed tentative appointment is charged a no-show fee
  given: {appointment: {status: tentative, starts_at: -1h}}
  do: NoShowSweepJob.perform_now
  expect: no PaymentGateway charge
- says: check-in between the sweep's load and its lock becomes a no-show
  given: {appointment: {status: booked, starts_at: -20m}}
  race: [NoShowSweepJob.perform_now, "CheckIn.new(appointment).call"]
  expect: appointment.status == "checked_in"

given makes one record per key from the factory of that name. do is Ruby. race holds the first action at a pause point while the second runs, then the reverse, and also runs both serial orders. expect is what must be true afterwards. The skill lists every form.

3. Check

bundle exec holdfast check suspicions.yml

The first run learns how the app runs its tests and makes test databases of its own. This is the whole output on the clinic app:

# holdfast

check: 2 suspicion(s); no change: checks run on the current tree only (uncommitted changes on 62b0f46); app, lib, config, db and the Gemfile match main (62b0f46)
kept specs: spec/holdfast/main/ (branch main); rows: .holdfast/check/main/check.jsonl
seconds: record 4.2, run 2.4

confirmed=2 could_not_run=0 refuted=0

## Confirmed (the kept spec fails)

- `a passed tentative appointment is charged a no-show fee`: no PaymentGateway charge does not hold (observed [[{customer: "pat_1", amount_cents: 2500, reference: "no-show-1"}]])
  - ran: NoShowSweepJob.perform_now returned [Appointment#1(status: "no_show")]
  - no PaymentGateway charge fails ([[{customer: "pat_1", amount_cents: 2500, reference: "no-show-1"}]])
  - spec output: suspicion confirmed: no PaymentGateway charge does not hold (observed [[{customer: "pat_1", amount_cents: 2500, reference: "no-show-1"}]])
  - repro: `DATABASE_URL=postgres://postgres@127.0.0.1:5440/hf_clinic_c5da1e_c2 TEST_ENV_NUMBER=c2 bundle exec rspec spec/holdfast/main/a_passed_tentative_appointment_is_charged_a_73c143_spec.rb`
  - next: keep spec/holdfast/main/a_passed_tentative_appointment_is_charged_a_73c143_spec.rb; it fails until the code is fixed
- `check-in between the sweep's load and its lock becomes a no-show`: first held at batch_load, second runs (held at Appointment load (1 rows)) ends [[Appointment#1(status: "no_show")], Appointment#1(status: "checked_in"); appointment.status == "checked_in" fails ("no_show")], an end no serial order reaches (first then second ends [[Appointment#1(status: "no_show")], raised CheckIn::Refused; appointment.status == "checked_in" fails ("no_show")]; second then first ends [[], Appointment#1(status: "checked_in"); appointment.status == "checked_in" holds ("checked_in")]); serial order "first then second" also fails expect, so this end is the evidence
  - pause points tried: batch_load: Appointment load (1 rows)
  - first held at batch_load, second runs: NoShowSweepJob.perform_now returned [Appointment#1(status: "no_show")]; CheckIn.new(appointment).call returned Appointment#1(status: "checked_in"); held at Appointment load (1 rows); appointment.status == "checked_in" fails ("no_show")
  - first then second: NoShowSweepJob.perform_now returned [Appointment#1(status: "no_show")]; CheckIn.new(appointment).call raised CheckIn::Refused: appointment is no_show; appointment.status == "checked_in" fails ("no_show")
  - second then first: CheckIn.new(appointment).call returned Appointment#1(status: "checked_in"); NoShowSweepJob.perform_now returned []; appointment.status == "checked_in" holds ("checked_in")
  - second held at batch_load, first runs: CheckIn.new(appointment).call returned Appointment#1(status: "checked_in"); NoShowSweepJob.perform_now returned [Appointment#1(status: "no_show")]; held at Appointment load (1 rows); the second waited on a lock; appointment.status == "checked_in" fails ("no_show")
  - spec output: suspicion confirmed: first held at batch_load, second runs (held at Appointment load (1 rows)) ends [[Appointment#1(status: "no_show")], Appointment#1(status: "checked_in"); appointment.status == "checked_in" fails ("no_show")], an end no serial order reaches (first then second ends [[Appointment#1(status: "no_show")], raised CheckIn::Refused; appointment.status == "checked_in" fails ("no_show")]; second then first ends [[], Appointment#1(status: "checked_in"); appointment.status == "checked_in" holds ("checked_in")]); serial order "first then second" also fails expect, so this end is the evidence
  - repro: `DATABASE_URL=postgres://postgres@127.0.0.1:5440/hf_clinic_c5da1e_c2 TEST_ENV_NUMBER=c2 bundle exec rspec spec/holdfast/main/check_in_between_the_sweep_s_load_and_its_lo_d80794_spec.rb`
  - next: keep spec/holdfast/main/check_in_between_the_sweep_s_load_and_its_lo_d80794_spec.rb; it fails until the code is fixed
rows: .holdfast/check/main/check.jsonl; kept specs: spec/holdfast/main/

The race is confirmed only because the held order ends in a state that no serial order reaches. A race whose end some serial order also reaches is refuted.

4. Keep

Each confirmed row names a spec under spec/holdfast/<branch>/ that fails until the code is fixed. Fix the code, run the spec, and keep it. After the sweep selects only booked appointments, the first spec passes:

bundle exec rspec spec/holdfast/main/a_passed_tentative_appointment_is_charged_a_73c143_spec.rb
1 example, 0 failures

Exit status 2 means a row was confirmed, 1 a usage error, and 0 that nothing was confirmed.

Check a change

On a branch that differs from the base branch (--base REF names another), each suspicion runs on base and on the working tree, both with head's factories, and the row says whether the change brought it in. Here the branch skip-tentative changes the sweep to select only booked appointments, and the same file runs again:

# holdfast

check: 2 suspicion(s); change: each runs on base main (62b0f46) and on the working tree (uncommitted changes on 7c510b0); 1 file(s) differ: app/jobs/no_show_sweep_job.rb
kept specs: spec/holdfast/skip-tentative/ (branch skip-tentative); rows: .holdfast/check/skip-tentative/check.jsonl
seconds: record 0.0, run 2.6

confirmed=1 could_not_run=0 refuted=1

## Confirmed (the kept spec fails)

- `check-in between the sweep's load and its lock becomes a no-show`: first held at batch_load, second runs (held at Appointment load (1 rows)) ends [...]
  - base 62b0f46: confirmed: [...]; it was there before the change
  [...]

## Refuted (what was observed)

- `a passed tentative appointment is charged a no-show fee`: no PaymentGateway charge holds (observed [])
  - base 62b0f46: confirmed: no PaymentGateway charge does not hold (observed [[{customer: "pat_1", amount_cents: 2500, reference: "no-show-1"}]]); the change removes it
  - ran: NoShowSweepJob.perform_now returned []
  - no PaymentGateway charge holds ([])
rows: .holdfast/check/skip-tentative/check.jsonl; kept specs: spec/holdfast/skip-tentative/

[...] marks text cut from the race row, which repeats the row above. A suspicion that calls a method, a column or a route that base does not have is confirmed on head only, and the row says that base cannot run it.

holdfast run checks a change without suspicions. It gives each changed method a row, and the callers of a changed constant or scope too, runs the tests that name each target on base and head, and drafts pins from each method's parameters. A row is unintended_change (passes on base, fails on head), intended_change (a difference you marked as meant, or new behavior that only the change's new specs run), unchecked (with the value nobody drew, to write as a suspicion), or clean (covering only the evidence it lists). This part runs on jitcatch and branchout, which come with holdfast.

Hunt when nothing changed

holdfast run with no change drafts one sequence definition per write area of the app (routes, Grape endpoints, jobs) into .holdfast/sequences/. It then runs random orders of their actions against the test database, under invariants, other time zones, two actions interleaved at an outside call, a job held after its first batch load, and an accepted request sent once more. A failure is shrunk to the shortest sequence, for at most 60 seconds. A connection to any host but this machine is refused, and the area reports the real service it reached; write a fake of that service in the area's file. --budget SECONDS caps the run (default 600). On the clinic app, with a budget of 120 seconds:

# holdfast

nothing in the app changed against main (b5c13d9)
seconds: record 4.6, hunt 40.8

unintended_change=0 bug_suspect=1 unchecked=0 intended_change=0 clean=0

## Suspected bugs

- `sequences: check_ins`: no_show_sweep_job loaded Appointment#1 with status "booked"; check_ins_create set it to "checked_in" while no_show_sweep_job held it; no_show_sweep_job then wrote "no_show", though its own query does not select a row whose status is "checked_in" (shrinking stopped after 30 s)
  - failure: no_show_sweep_job loaded Appointment#1 with status "booked"; check_ins_create set it to "checked_in" while no_show_sweep_job held it; no_show_sweep_job then wrote "no_show", though its own query does not select a row whose status is "checked_in" (shrinking stopped after 30 s)
  - log: .holdfast/runs/sequences/check_ins.log (the shortest sequence and an `it` block that replays it)
  - next: paste the `it` block from the log into .holdfast/sequences/check_ins.rb, see it fail, fix the code, run again
rows: .holdfast/verdicts.jsonl

The hunt found the same race as the second suspicion, with no suspicion written. A hunt row leads to a suspicion: write it down and confirm it with holdfast check. holdfast run --sequences also hunts over the areas a change touches, and --observe runs each area's action orders on base and head and reports each step that differs.

Usage

holdfast check [FILE]... [--suspect YAML]... [--base REF] [--here] [--repo DIR]
holdfast run [--base REF] [--target NAME]... [--only NAME]... [--intent TEXT] [--learn] [--sequences] [--observe] [--budget SECONDS] [--repo DIR]
holdfast show
  • --suspect YAML passes one suspicion as a YAML flow mapping, and repeats.
  • --here runs a check once on the working tree, even in a change.
  • --learn learns again how the app runs its tests, after you fix a failed setup.
  • show prints the last summary of holdfast run again.

holdfast writes under .holdfast/ and spec/holdfast/ in the app. Add .holdfast/ to the app's .gitignore. Commit the specs you keep.

Each app gets its own test database names through DATABASE_URL, made from its directory, so base and head, and two apps on one server, do not share one. On SQLite, each side gets a file of its own under .holdfast/db/, and the app's own test database file is never written. The setup runs bin/rails db:create db:schema:load.

Races and the hunt run on PostgreSQL and on SQLite. SQLite lets one connection write at a time: when a race holds an action that has already written in its open transaction, the other action waits for it, and the row says the second waited on a lock. MySQL is not tested.

Limits

  • A drafted pin whose inputs build no record checks nothing, and holdfast run can still list its target as clean when the pin loads the changed line. Read the evidence of a clean row before you trust it.
  • A race is tried at the pause points batch_load, outside_call, row_lock, first_write, or a method you name. Other interleavings are not tried.
  • The hunt reaches only the actions whose drafted setup and sign-in work. The first line says how many it did not reach.

Development

bundle install
bundle install --gemfile test/fixture/clinic/Gemfile
DB_HOST=localhost DB_PORT=5432 ruby -Ilib test/run.rb
bundle exec rbs-inline --output=sig/generated lib
bundle exec steep check

Without DB_HOST, the tests that run the clinic app skip. The last two commands write the RBS signatures in sig/generated/ from the type comments in lib/, and check lib/ against them. AGENTS.md has the details.

License

MIT


Japanese

About

Run what you suspect in a Rails app on base and head, and keep the spec that proves it.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages