v1.60.0.0 feat(learnings): signal-gated capture + helpful/harmful counters - #2030
Open
tonyjzhou wants to merge 5 commits into
Open
v1.60.0.0 feat(learnings): signal-gated capture + helpful/harmful counters#2030tonyjzhou wants to merge 5 commits into
tonyjzhou wants to merge 5 commits into
Conversation
Opt-in --signal routes a lesson to the trusted store only when a reliable in-session signal fired (tests-passed/app-ran-clean/validator/benchmark/ exec-success) or source=user-stated; otherwise it parks in learnings-candidates.jsonl with confidence ceiling-clamped to 4. Without --signal, historical behavior is preserved (everything trusted).
New append-only, event-sourced gstack-learnings-feedback records helpful/harmful events. gstack-learnings-search now ranks by decayed confidence PLUS net feedback, flags net-negative entries for prune, and adds --candidates to list the gated-out pool. No feedback present => identical ranking to before.
…ls + /learn The capture block every tier-2 skill emits now passes --signal; the search block prompts feedback when a learning is applied and a session ends green. Adds /learn candidates (promote) and /learn feedback. Regenerated SKILL.md across hosts and refreshed the codex/factory ship golden baselines.
11 tests covering trusted-vs-candidate routing (incl. confidence cap and backward compat), the candidates pool, append-only feedback events, and the reinforced ranking with prune flagging.
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
gstack-learnings-searchhas ranked learnings by a self-assigned confidence decayed only byage. So an unverified hunch landed in the trusted store next to a test-verified truth, and a
lesson proven right many times decayed at the same rate as a spurious one. This adds the two
missing halves: capture is gated on a real signal, and ranking rewards what proves out.
Signal-gating (capture)
gstack-learnings-loggains opt-in--signal. A lesson backed by an objective check thatfired this session (
tests-passed,app-ran-clean,validator,benchmark,exec-success)or stated by the user goes to the trusted store.
--signal noneparks it inlearnings-candidates.jsonlwith confidence ceiling-clamped to 4, so an unverified guess canno longer outrank a verified lesson. Without
--signal, behavior is unchanged, so nothingbreaks on day one.
Counters (ranking)
New
gstack-learnings-feedbackrecords, append-only and event-sourced, that a learning helpedor misled you.
gstack-learnings-searchnow ranks by decayed confidence PLUS net feedback, soproven lessons rise above their stated confidence and net-negative ones sink and get flagged
for prune. Counters never touch the learning row; the append-only log stays the source of truth.
Wiring
The capture and search blocks every tier-2 skill emits (
scripts/resolvers/learnings.ts) nowpass
--signaland prompt feedback when a learning is applied and a session ends green. New/learn candidatespromotes parked lessons;/learn feedbackrecords by hand.Tests
test/gstack-learnings-reinforce.test.tsadds 11 tests: trusted-vs-candidate routing(confidence cap, backward compat, user-stated-always-trusted, invalid-signal rejection), the
candidates pool, append-only feedback, and reinforced ranking with prune flagging. The existing
gstack-learnings-searchsuite still passes (changes are additive). Golden ship baselines(codex, factory) were refreshed for the new blocks, and the resolver additions were kept tight
so office-hours stays under the parity ratchet.
The free unit tier is green for everything this touches. The remaining failures in a full local
bun test test/run are pre-existing and environmental (brain-cache schema tests,resolve-user-slug sandbox timeouts, an iOS XCTest-toolchain test, a hook-install test); they
fail identically on
main(c7ae632) without this branch.Numbers
On the test fixtures: an unverified guess (conf 9, no signal) is parked at 4 instead of trusted
at 9; a learning with +2 net feedback ranks at 11, above the 10 cap; one with -2 sinks to 7 and
is flagged PRUNE. With no signals and no feedback, the store ranks exactly as today. Reproduce:
bun test test/gstack-learnings-reinforce.test.ts.VERSION 1.58.1.0 -> 1.60.0.0 (1.59.0.0 is claimed by the open learnings-dedup PR #2029).
Note for CI
This is a fork PR, so the eval and E2E jobs will not receive base-repo secrets and will fail on
empty-env auth. The free unit tier runs without secrets. Happy to re-push to a base-repo branch
if you would rather run the full eval suite.