Skip to content

agent: scheduled verify and durability-refresh cadences#172

Merged
mbertschler merged 8 commits into
mainfrom
claude/issue-154-verify-durability-cadences
Jul 25, 2026
Merged

agent: scheduled verify and durability-refresh cadences#172
mbertschler merged 8 commits into
mainfrom
claude/issue-154-verify-durability-cadences

Conversation

@mbertschler

Copy link
Copy Markdown
Owner

Summary

Closes the two standing automation-loop gaps from the friction log (F32, F33)
so a correctly configured household keeps its offsite evidence fresh with zero
typed commands — a prerequisite for the fingerprint-verified offload upgrade
that stacks on this (#155).

  • F32 — scheduled verify. A verify_every cadence per content-addressed /
    packed destination, plus an [agent]-level default, drives the same pass
    as squirrel verify and records it as an audit run exactly like the manual
    command. Offsite bitrot detection no longer depends on anyone typing it.
  • F33 — scheduled durability refresh. A pull_durability_every cadence per
    [nodes.X] gives relayed durability evidence its own clock, independent of
    sync activity. A receive-only node (the reference htpc) now refreshes its
    offload-gate evidence unattended, and any node keeps refreshing even when
    nothing changes — exactly when offload_max_evidence_age starts counting.

Both surface in the scheduler log with the existing kicked/finished/error
discipline and respect the marker/init boundary: verify is read-only, the
durability pull never rewinds a watermark (allowRewind is always false), and
neither ever bootstraps a destination — the agent does not escalate.

Changes

  • config: verify_every on destinations (rejected on non-verifiable
    layouts) and on [agent] (fleet default); pull_durability_every on
    [nodes.X]. All reuse the existing cadence-floor parsing.
  • agent/scheduler: two new non-volume evaluation phases keyed on
    destinations and peer nodes, with in-memory cadence watermarks. The start
    gate is broadened (anyScheduledWork) so a receive-only node with only a
    pull cadence still runs the scheduler. Verify/pull are injected as
    VerifyRunner / DurabilityPuller closures so the agent package keeps no
    import edge to sync.
  • cmd/squirrel/agent: wires the closures over sync.VerifyRemote and
    sync.PullDurability; rclone is resolved for a verify-only schedule too.
  • store: BeginDurabilityPullRun (shares a helper with
    BeginRemoteVerifyRun) records the pull as a volume-less audit run with a
    new pull-durability runs_audit transition — so it appears in squirrel runs yet stays out of the per-volume drift-audit reads.
  • docs / design: reference-setup steady-state table + nas/htpc configs now
    exercise both cadences; configuration reference and agent guide document them.

No schema changekind='audit' and free-text runs_audit transitions
already exist (issue #157 owns the v25 bump in parallel).

Design decision to double-check

Durability-pull runs are recorded with volume_id NULL (mirroring
BeginRemoteVerifyRun), which keeps them out of ListAuditRunsSince (the
drift-since-last-sync handshake) and the TUI's per-volume "last audit" — the
pulled volume/peer live in the runs_audit note instead. The cadence watermarks
are in-memory rather than persisted: both operations are cheap, idempotent, and
read-only w.r.t. content, so re-running once after an agent restart is harmless
and (for evidence freshness) desirable.

Testing

  • go vet ./... — clean
  • go test ./... — all packages pass
  • gofmt -l — clean
  • New tests: config parsing for all three knobs (incl. rejections); scheduler
    verify/durability cadence walks (due/skip/re-fire, no-op, error-consumes-
    cadence, stake filter, partial-on-rewind, nil-runner inert, broadened start
    gate); BeginDurabilityPullRun shape + handshake-isolation; CLI status
    mappers and wiring predicates.
  • golangci-lint could not run locally (the sandbox's linter binary is built
    with go1.25 and refuses this module's go 1.26.1 directive); CI will run it.

Unblocks #155.

Closes #154

🤖 Generated with Claude Code

https://claude.ai/code/session_014Skm2Qm6wWUB7dD4CpPTz7


Generated by Claude Code

Close the two standing automation-loop gaps (friction log F32/F33) so a
correctly configured household keeps its offsite evidence fresh with zero
typed commands.

- verify_every: a per-destination cadence (content-addressed/packed only)
  plus an [agent]-level default, driving the same pass as `squirrel verify`
  and recording an `audit` run exactly like the manual command.
- pull_durability_every: a per-node cadence that gives relayed durability
  evidence its own clock, independent of sync activity. A receive-only node
  (the reference htpc) now refreshes its offload-gate evidence unattended.

Both surface in the scheduler log with the existing kicked/finished/error
discipline and respect the marker/init boundary: verify is read-only, the
durability pull never rewinds a watermark (allowRewind is always false), and
neither ever bootstraps a destination — the agent does not escalate.

Plumbing: the scheduler gains two non-volume evaluation phases keyed on
destinations and peer nodes, with in-memory cadence watermarks (cheap,
idempotent, read-only work re-run once after a restart is harmless and, for
evidence freshness, desirable). The verify/pull operations are injected as
VerifyRunner/DurabilityPuller closures so the agent package keeps no edge to
sync. Durability-pull runs are recorded as volume-less `audit` runs (like
BeginRemoteVerifyRun), keeping them out of the per-volume drift-audit reads
while still appearing in `squirrel runs`.

No schema change: kind='audit' and free-text runs_audit transitions already
exist. Docs and the reference setup are updated to configure and describe
both cadences.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Skm2Qm6wWUB7dD4CpPTz7
Copilot AI review requested due to automatic review settings July 24, 2026 11:16

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds two new agent scheduler cadences to close automation gaps: periodic remote verification for content-addressed/packed destinations (verify_every) and periodic peer durability refresh (pull_durability_every) so evidence freshness doesn’t depend on manual commands or sync activity.

Changes:

  • Config: add verify_every (per-destination and [agent] default) and pull_durability_every (per-node) with validation and tests.
  • Agent/scheduler: add destination-scoped verify and (volume,peer)-scoped durability-pull evaluation phases, including deterministic logging and in-memory cadence watermarks.
  • Store/docs/CLI wiring: record durability pulls as volume-less audit runs with a new pull-durability runs_audit transition; wire agent closures to sync.VerifyRemote and sync.PullDurability; document new knobs and update reference setup.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
store/runs.go Factor out a shared helper for volume-less audit runs; add BeginDurabilityPullRun.
store/runs_audit.go Add TransitionPullDurability audit transition constant.
store/remote_objects_test.go Add test asserting durability-pull runs are volume-less and excluded from per-volume audit reads.
docs/src/content/docs/reference/configuration.md Document verify_every (dest + agent default) and pull_durability_every (node).
docs/src/content/docs/guides/agent.md Document scheduled verify and scheduled durability pull behavior.
design/reference-setup.md Update reference household config/examples to exercise both new cadences.
config/nodes.go Parse and expose pull_durability_every on nodes.
config/destinations.go Parse/validate per-destination verify_every and reject it for non-verifiable layouts.
config/config.go Add Destination.VerifyEvery to resolved config.
config/config_test.go Add tests for destination verify cadence, agent default verify cadence, and node pull cadence parsing/errors.
config/agent.go Parse and expose [agent] verify_every default cadence.
cmd/squirrel/agent.go Wire scheduler closures for verify/durability-pull; broaden rclone setup to verify-only schedules; add run bookkeeping for durability pulls.
cmd/squirrel/agent_scheduler_test.go Add unit tests for mapping/status helpers and wiring predicates.
agent/serve.go Start scheduler loop when any scheduled work exists (volume, verify, or pull).
agent/scheduler.go Add verify and durability-pull cadence evaluation phases with in-memory watermarks and deterministic ordering.
agent/scheduler_test.go Update scheduler fixture to initialize new cadence fields/watermarks.
agent/scheduler_cadence_test.go Add cadence-walk tests for verify and durability pulls (due/skip/refire/no-op/error/rewind/nil-runner/start-gate).
agent/agent.go Extend agent config and public types for injected verify runner and durability puller.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread store/runs.go Outdated
Comment thread cmd/squirrel/agent.go
claude added 2 commits July 24, 2026 11:23
- store: BeginDurabilityPullRun now labels its run "pull-durability",
  matching the runs_audit transition constant and the scheduler's log
  kind so the run, its note, and the scheduler line all correlate under
  one name (was "durability-pull").
- cmd/squirrel/agent: finishDurabilityPullRun joins the pull error with
  the AppendRunAudit/FinishRun bookkeeping errors (errors.Join) instead
  of only surfacing a bookkeeping error when the pull itself succeeded.
  A failure to record or finish the audit run — which would otherwise
  strand a 'running' row — now always reaches the scheduler error log.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Skm2Qm6wWUB7dD4CpPTz7
Make offload reachable against packed/CA offsites by fixing the two
friction-log defects that together closed every gate path (F13 + F29).

F13 — the pending-artifact gate was per-run. certifyPacked held the
vector only when *this run's* packs were unverified, so a later run that
packed nothing advanced the whole volume past an earlier still-pending
pack, and once `squirrel verify` filled the fingerprints nothing
re-advanced. The advance is now gated on zero pending artifacts for the
whole (volume, destination) pair (CountVolumeContentsPendingFingerprint),
and `squirrel verify` re-attempts the advance itself once a clean pass
completes (UpgradeDestinationVectorToFingerprintVerified).

F29 — the component method was never upgraded past presence+size, so the
fail-closed gate refused relayed offsites forever. Capture and verify now
upgrade a fully fingerprint-verified pair to a new `fingerprint-verified`
verify_method (a free-text value; no schema change, no CHECK constraint).
It relays over the durability pull like any other method.

Decision 1b — a verified provider-fingerprint chain counts as
content-verified for the gate, but only while a scheduled verify cadence
(#154) keeps re-confirming it. The responder relays its effective verify
cadence per destination (DurabilityComponent.VerifyEveryNs); the pull
bakes a relayed fingerprint-verified component down to presence+size when
that cadence is absent (older peer, or a destination the responder runs
on no cadence), failing closed. The gate accepts a fingerprint-verified
component when relayed (cadence proven at pull time) or when the local
destination carries a live cadence; a local component without a cadence
falls back to the per-object/pack scan-back, since the node re-reads its
own fingerprints — the coupling binds the relayed path, where the
evidence cannot be re-checked locally.

Tests: both F13 sides (per-state hold, verify-then-advance) for packed
and content-addressed, the capture-time method upgrade, the relayed
accept / relayed-absent refuse cadence coupling, the local
cadence/fallback/refuse cases, the pull-time baking, and the responder
relaying its cadence.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Skm2Qm6wWUB7dD4CpPTz7
claude and others added 5 commits July 24, 2026 12:04
The comment implied the fingerprint pending/advance gate covers the
placement map and manifest segment, but CountVolumeContentsPendingFingerprint
only considers the volume's present file contents (verified via
remote_objects / remote_packs). Reword to state exactly what the gate
covers — packs and per-hash objects, the artifacts that carry content bytes
— and note that the placement map and manifest segment are re-derivable
metadata whose landing is confirmed upstream in push (presence+size, before
the run is promoted to success), so a run whose map or segment failed to
land never reaches certifyPacked. Comment-only; no behaviour change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Skm2Qm6wWUB7dD4CpPTz7
The scheduler's pull-durability finished line logged fetched/applied/
dropped but omitted the rewinds count, leaving DurabilityPullReport.Rewinds
set (cmd wiring) yet never read — an unused field on a public type, and an
observability gap: a refused rewind surfaced only as status=partial with no
count. Refused rewinds are a first-class "scary moment" (ux-principle 4) and
automation is never invisible (ux-principle 5), so the count belongs in the
finished line alongside dropped. Test now asserts rewinds=1 appears.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Skm2Qm6wWUB7dD4CpPTz7
Textual conflicts:
- cmd/squirrel/agent.go: keep this branch's agent.Config wiring (Nodes,
  VerifyEvery, VerifyRunner, DurabilityPuller) on top of main's `version`
  var, which replaced this branch's local `agentVersion` const.
- cmd/squirrel/offload.go: both sides appended a different helper at the
  same spot and happened to share a loop-closing tail — keep this
  branch's verifyCadencedTargets and main's relayed-capability
  pre-check helpers (#145), each with its own tail.
- agent/durability_test.go: git interleaved two independent endpoint
  tests that share the request boilerplate — this branch's relayed
  verify-cadence assertions and main's capability advertisement. Split
  back into two functions.

Semantic conflict: both sides added a Destinations field to agent.Config,
for different readers — main's durability endpoint advertising gating
capability, this branch's scheduler finding verify cadences. Same name,
same type, same resolved map; collapsed into one field whose comment
documents both consumers.

go vet, go test ./..., and gofmt are clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQUzRmQHqbP2kV9kFrbBHL
main now carries #171 and the listener-less agent mode (#175), which is
where this branch's two new cadences had to be reconciled.

Textual conflicts:
- store/runs_audit.go, agent/agent.go: both sides appended at the same
  spot — keep this branch's pull-durability transition and its
  VerifyRunner/DurabilityPuller types alongside main's reset-destination
  transition; main rewrote the Config doc comment for listener-less mode,
  so its version of that comment stands.
- docs reference: both sides added to the [agent] section. Kept main's
  listener-less paragraph and this branch's key tables, and corrected the
  table to match — `listen` is no longer required, and `auth.token` is
  required only alongside it. Left as-is the two would have contradicted
  each other in adjacent paragraphs.

Semantic conflict: main's listener-less start gate (cmd's agentHasWork)
duplicated the scheduler's volume-cadence logic and predated this branch's
verify_every / pull_durability_every, so it would have refused to start a
listener-less agent whose only work is one of those cadences — exactly the
receive-only htpc in design/reference-setup.md that F33 exists to serve.
Replaced the duplicate with an exported agent.ScheduledWorkInConfig, built
on the same resolvers the scheduler's own anyScheduledWork gate uses, so
the two cannot drift again; the refusal message now names every cadence.
Covered by a table test including the receive-only pull-only config.

go vet, go test ./..., and gofmt are clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQUzRmQHqbP2kV9kFrbBHL
@mbertschler
mbertschler merged commit f3df460 into main Jul 25, 2026
3 checks passed
@mbertschler
mbertschler deleted the claude/issue-154-verify-durability-cadences branch July 25, 2026 08:09
mbertschler pushed a commit that referenced this pull request Jul 25, 2026
main now carries #172's verify and durability-refresh cadences.

Textual conflicts:
- agent/scheduler.go, agent/scheduler_test.go: this branch replaced the
  scheduler's syncRun field with the dispatcher; main added the
  verify/pull cadence fields beside it. Keep the dispatcher and graft the
  cadence fields on.
- cmd/squirrel/agent.go: keep the startup reaper and drop agentHasWork —
  main replaced it with the shared agent.ScheduledWorkInConfig this file
  already calls. In resolveSchedulerRclone, keep the stall-timeout guard
  and put main's sync-only gate around the version preflight (a
  verify-only schedule reads provider checksums and needs no `--hash
  blake3` check, but every automatic transfer still wants the guard).
- store/runs_audit.go: both sides added transitions; keep all of them.

Semantic conflict: #172 gave loadGate a `cadenced` argument, which this
branch's offload.Readiness (added for the status surface) did not pass —
the two files merged cleanly and stopped compiling. Readiness must feed
the gate exactly what a real offload feeds it: `cadenced` widens the gate
for locally-advanced fingerprint-verified components, so omitting it
would make "N GB offloadable now" under-report against what an offload
would actually move — the precise contract Readiness documents.

The set is now needed in two packages, so the resolver moved out of
cmd/squirrel to config.Config.VerifyCadencedTargets, beside the
CanEverGateOffload predicate it complements; the offload command and the
status readiness query now share one implementation.

go vet, go test ./..., and gofmt are clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQUzRmQHqbP2kV9kFrbBHL
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

agent: scheduled verify and durability-refresh cadences

3 participants