Skip to content

v1.60.0.0 feat(learnings): signal-gated capture + helpful/harmful counters - #2030

Open
tonyjzhou wants to merge 5 commits into
garrytan:mainfrom
tonyjzhou:learnings-reinforce
Open

v1.60.0.0 feat(learnings): signal-gated capture + helpful/harmful counters#2030
tonyjzhou wants to merge 5 commits into
garrytan:mainfrom
tonyjzhou:learnings-reinforce

Conversation

@tonyjzhou

Copy link
Copy Markdown

What

gstack-learnings-search has ranked learnings by a self-assigned confidence decayed only by
age. So an unverified hunch landed in the trusted store next to a test-verified truth, and a
lesson proven right many times decayed at the same rate as a spurious one. This adds the two
missing halves: capture is gated on a real signal, and ranking rewards what proves out.

Signal-gating (capture)

gstack-learnings-log gains opt-in --signal. A lesson backed by an objective check that
fired this session (tests-passed, app-ran-clean, validator, benchmark, exec-success)
or stated by the user goes to the trusted store. --signal none parks it in
learnings-candidates.jsonl with confidence ceiling-clamped to 4, so an unverified guess can
no longer outrank a verified lesson. Without --signal, behavior is unchanged, so nothing
breaks on day one.

Counters (ranking)

New gstack-learnings-feedback records, append-only and event-sourced, that a learning helped
or misled you. gstack-learnings-search now ranks by decayed confidence PLUS net feedback, so
proven lessons rise above their stated confidence and net-negative ones sink and get flagged
for prune. Counters never touch the learning row; the append-only log stays the source of truth.

Wiring

The capture and search blocks every tier-2 skill emits (scripts/resolvers/learnings.ts) now
pass --signal and prompt feedback when a learning is applied and a session ends green. New
/learn candidates promotes parked lessons; /learn feedback records by hand.

Tests

test/gstack-learnings-reinforce.test.ts adds 11 tests: trusted-vs-candidate routing
(confidence cap, backward compat, user-stated-always-trusted, invalid-signal rejection), the
candidates pool, append-only feedback, and reinforced ranking with prune flagging. The existing
gstack-learnings-search suite still passes (changes are additive). Golden ship baselines
(codex, factory) were refreshed for the new blocks, and the resolver additions were kept tight
so office-hours stays under the parity ratchet.

The free unit tier is green for everything this touches. The remaining failures in a full local
bun test test/ run are pre-existing and environmental (brain-cache schema tests,
resolve-user-slug sandbox timeouts, an iOS XCTest-toolchain test, a hook-install test); they
fail identically on main (c7ae632) without this branch.

Numbers

On the test fixtures: an unverified guess (conf 9, no signal) is parked at 4 instead of trusted
at 9; a learning with +2 net feedback ranks at 11, above the 10 cap; one with -2 sinks to 7 and
is flagged PRUNE. With no signals and no feedback, the store ranks exactly as today. Reproduce:
bun test test/gstack-learnings-reinforce.test.ts.

VERSION 1.58.1.0 -> 1.60.0.0 (1.59.0.0 is claimed by the open learnings-dedup PR #2029).

Note for CI

This is a fork PR, so the eval and E2E jobs will not receive base-repo secrets and will fail on
empty-env auth. The free unit tier runs without secrets. Happy to re-push to a base-repo branch
if you would rather run the full eval suite.

Opt-in --signal routes a lesson to the trusted store only when a reliable
in-session signal fired (tests-passed/app-ran-clean/validator/benchmark/
exec-success) or source=user-stated; otherwise it parks in learnings-candidates.jsonl
with confidence ceiling-clamped to 4. Without --signal, historical behavior is
preserved (everything trusted).
New append-only, event-sourced gstack-learnings-feedback records helpful/harmful
events. gstack-learnings-search now ranks by decayed confidence PLUS net feedback,
flags net-negative entries for prune, and adds --candidates to list the gated-out
pool. No feedback present => identical ranking to before.
…ls + /learn

The capture block every tier-2 skill emits now passes --signal; the search block
prompts feedback when a learning is applied and a session ends green. Adds /learn
candidates (promote) and /learn feedback. Regenerated SKILL.md across hosts and
refreshed the codex/factory ship golden baselines.
11 tests covering trusted-vs-candidate routing (incl. confidence cap and backward
compat), the candidates pool, append-only feedback events, and the reinforced
ranking with prune flagging.
@trunk-io

trunk-io Bot commented Jun 16, 2026

Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant