Add distill, layered context loading, and smarter retrieval

.brain/brainstorms/mempalace-inspired-brain-improvements.md

@@ -0,0 +1,50 @@
+---
+brainstorm_status: active
+created: "2026-04-13T06:23:51Z"
+idea_count: 0
+project: brain
+title: MemPalace-Inspired Brain Improvements
+type: brainstorm
+updated: "2026-04-13T06:23:51Z"
+---
+# Brainstorm: MemPalace-Inspired Brain Improvements
+
+Started: 2026-04-13T06:27:00Z
+
+## Focus Question
+
+Which MemPalace-style ideas are actually worth borrowing for Brain without breaking Brain's explicit, curated-memory philosophy?
+
+## Ideas
+
+- Optional session-aware memory distillation can recover decisions, tradeoffs, bugs, and discoveries that currently disappear when nobody writes them down.
+- Distillation should propose note updates, not silently write them. Brain keeps human or agent approval in the loop.
+- Layered context loading can split startup context into L0 identity/current-state, L1 workflow and architecture summary, L2 full context files, and L3 search-driven deep context.
+- Retrieval ranking should favor what matters now, not just literal query overlap. Recency, note type, and active epic or spec context should affect ordering.
+- Search should bridge directly into usage with an inject mode that returns a preformatted context block an agent can paste into work.
+- Brain needs first-class decision notes so the system keeps why a choice was made, not just the final choice.
+- Session closeout should help the user recover missing memory work by suggesting a session-scoped distill path when repo state changed but durable notes did not.
+- A lightweight session memory cache is interesting only as an optional temporary layer. It should stay clearly separate from durable memory until distilled.
+
+## Related
+
+- [[.brain/planning/epics/planning-and-brainstorming-ux.md]]
+- [[.brain/planning/epics/retrieval-and-index-lifecycle.md]]
+- [[.brain/planning/epics/context-and-session-workflow.md]]
+
+## Raw Notes
+
+High-value changes from the brainstorm:
+
+1. Memory capture: add an optional `brain distill` flow that inspects recent session activity, command runs, and git diff, then proposes durable note updates.
+2. Context efficiency: add layered context loading so agents can start small and pull deeper context only when needed.
+3. Retrieval quality: improve ranking with recency, note-type weighting, and active-work-context bias.
+4. Reasoning preservation: add a first-class decision note type and template.
+5. Workflow smoothing: integrate distillation suggestions into session closeout instead of only blocking on missing memory updates.
+6. Retrieval-to-usage bridge: add `brain search --inject` or equivalent agent-ready context output.
+
+Non-goals from the same brainstorm:
+
+- Do not store everything verbatim.
+- Do not introduce spatial memory abstractions that duplicate the filesystem.
+- Do not lean harder on embeddings at the cost of explicit project structure.

.brain/context/architecture.md

@@ -10,6 +10,7 @@ Use this file for the structural shape of the repository.
 - `internal/brainstorm/`
 - `internal/buildinfo/`
 - `internal/config/`
+- `internal/distill/`
 - `internal/embeddings/`
 - `internal/history/`
 - `internal/index/`

.brain/context/current-state.md

@@ -1,5 +1,5 @@
 ---
-updated: "2026-04-12T05:22:46Z"
+updated: "2026-04-13T15:26:30Z"
 ---
 # Current State
 
@@ -12,9 +12,10 @@ This file is a deterministic snapshot of the repository state at the last refres
 - Root: `.`
 - Runtime: `go`
 - Go module: `brain`
-- Current branch: `docs-pr-workflow-and-windows-ci-fixes`
+- Current branch: `feature/mempalace-planning-roadmap`
+- Default branch: `main`
 - Remote: `https://github.com/JimmyMcBride/brain.git`
-- Go test files: `19`
+- Go test files: `20`
 
 ## Docs
 
@@ -78,3 +79,10 @@ Add repo-specific notes here. `brain context refresh` preserves content outside
 - 2026-04-11: Repositioned the product docs around Brain as durable local operating memory for AI coding agents. The README now leads with the continuity/reliability story, keeps install + skill install + brainstorm-to-execution prominent, treats lower token and tool cost as supporting proof, and replaces the bottom link dump with high-level overview sections that deep-link into usage, architecture, skills, and why.
 - 2026-04-11: Tightened the Windows session-lock fix again after PR feedback. Lock acquisition now treats Windows `os.ErrPermission` during lock-directory races as retryable contention even if a follow-up stat misses the directory, which should stop intermittent `Access is denied` failures in concurrent session tests.
 - 2026-04-11: Tightened the README positioning pass by adding an explicit audience section for heavy AI-agent users, a `Why Brain Exists` founder-pain section, and a cleaner deep-dive section structure so the README sells urgency before dropping into technical overviews.
+- 2026-04-12: Returned the local checkout to `main` after PR #5 merged and fast-forwarded the repo to `v0.1.7`, which includes the README urgency follow-up and aligned supporting docs.
+
+- 2026-04-13: Updated `skills/brain/SKILL.md` and refreshed the global Codex `brain` skill so it now teaches `brain distill`, `brain search --inject`, `brain context load --level`, and the session-finish distill recovery flow.
+
+- 2026-04-13: Fixed the CLI test output normalizer so Windows no longer corrupts JSON escape sequences while replacing temp-root paths, which restores `TestCLIContextLoadLevels` and other JSON-based CLI tests on the Windows pipeline.
+
+- 2026-04-13: Fixed the remaining Windows CLI skill-target assertions to use OS-native path joins in tests, so `TestCLISkillsCommands` now accepts Windows `\` paths without regressing Unix output expectations.

.brain/context/workflows.md

@@ -15,11 +15,24 @@ Use this file for agent operating workflow inside the repo.
 - Keep durable discoveries, decisions, and risks in AGENTS.md, /docs, or .brain notes.
 - Update existing durable notes instead of duplicating context.
 - Run required verification commands through `brain session run -- <command>`.
+- If you change Brain command behavior or agent-facing workflow guidance, update `skills/brain/SKILL.md` in the same branch.
 - Re-read context before large changes if the task shifts.
 
+## Ticket Loop
+
+1. Start one story or ticket at a time and keep the scope narrow.
+2. Implement the story, then run focused tests for the touched packages.
+3. Run the required full checks through `brain session run -- go test ./...` and `brain session run -- go build ./...`.
+4. Review the diff against the story acceptance criteria and user-facing behavior.
+5. If review finds issues, patch the work and repeat the test and review steps.
+6. When the story is clean, commit it, push it, and only then move to the next story.
+
 ## Close-Out
 
 - Refresh or update durable notes for meaningful behavior, config, or architecture changes.
+- If `skills/brain/` changed, reinstall the local Brain skill for Codex and OpenClaw with `brain skills install --scope local --agent codex --agent openclaw --project .`.
+- When opening a PR, make the title and body release-note friendly because GitHub release notes are generated from merged PR metadata.
+- Summarize shipped behavior in the PR, not just implementation steps, so future changelogs stay human-readable.
 - Finish with `brain session finish`.
 - If you must bypass enforcement, use `brain session finish --force --reason "..."` so the override is recorded.
 <!-- brain:end context-workflows -->

.brain/planning/epics/context-and-session-workflow.md

@@ -4,18 +4,26 @@ project: brain
 spec: context-and-session-workflow
 title: Context And Session Workflow
 type: epic
-updated: "2026-04-11T21:53:09Z"
+updated: "2026-04-13T06:23:51Z"
 ---
 # Context And Session Workflow
 
 Created: 2026-04-11T05:10:54Z
 
 ## Description
 
+Make Brain's context loading and session closeout flow more graduated and more helpful. Agents should be able to start with less context, pull more intentionally, and recover from missing memory work without leaving the session workflow.
+
 ## Work Items
 
+- [[.brain/planning/stories/context-and-session-workflow-current-state-and-next-actions.md]]
+- [[.brain/planning/stories/add-layered-context-loading-levels.md]]
+- [[.brain/planning/stories/suggest-distillation-during-session-closeout.md]]
+
 ## Notes
 
+This epic is the workflow glue around context tiers and session-to-memory recovery. Temporary session cache work stays optional until the core path exists.
+
 ## Spec
 
 - [[.brain/planning/specs/context-and-session-workflow.md]]

.brain/planning/epics/planning-and-brainstorming-ux.md

@@ -4,18 +4,26 @@ project: brain
 spec: planning-and-brainstorming-ux
 title: Planning And Brainstorming UX
 type: epic
-updated: "2026-04-11T21:53:09Z"
+updated: "2026-04-13T06:23:51Z"
 ---
 # Planning And Brainstorming UX
 
 Created: 2026-04-11T05:10:54Z
 
 ## Description
 
+Close the gap between useful work happening in sessions and the durable project memory Brain relies on. Add optional distillation and reasoning capture without abandoning explicit, reviewed markdown as the source of truth.
+
 ## Work Items
 
+- [[.brain/planning/stories/planning-and-brainstorming-ux-current-state-and-next-actions.md]]
+- [[.brain/planning/stories/add-session-aware-memory-distillation.md]]
+- [[.brain/planning/stories/capture-decision-rationale-as-first-class-notes.md]]
+
 ## Notes
 
+This epic is about memory capture quality, not transcript storage. The approval loop stays intact.
+
 ## Spec
 
 - [[.brain/planning/specs/planning-and-brainstorming-ux.md]]

.brain/planning/epics/retrieval-and-index-lifecycle.md

@@ -4,18 +4,26 @@ project: brain
 spec: retrieval-and-index-lifecycle
 title: Retrieval And Index Lifecycle
 type: epic
-updated: "2026-04-11T21:53:09Z"
+updated: "2026-04-13T06:23:51Z"
 ---
 # Retrieval And Index Lifecycle
 
 Created: 2026-04-11T05:10:54Z
 
 ## Description
 
+Improve retrieval so Brain ranks and packages context around the work that matters now. Focus on better ranking signals and an agent-ready search handoff while keeping the local markdown-first index model intact.
+
 ## Work Items
 
+- [[.brain/planning/stories/retrieval-and-index-lifecycle-current-state-and-next-actions.md]]
+- [[.brain/planning/stories/improve-retrieval-ranking-with-recency-and-active-context.md]]
+- [[.brain/planning/stories/turn-search-results-into-agent-ready-context-blocks.md]]
+
 ## Notes
 
+The point is to improve retrieval quality and usability, not to add a more magical storage model.
+
 ## Spec
 
 - [[.brain/planning/specs/retrieval-and-index-lifecycle.md]]

.brain/planning/specs/context-and-session-workflow.md

@@ -2,35 +2,85 @@
 created: "2026-04-11T21:53:09Z"
 epic: context-and-session-workflow
 project: brain
-status: draft
+status: approved
 title: Context And Session Workflow Spec
 type: spec
-updated: "2026-04-11T21:53:09Z"
+updated: "2026-04-13T06:23:52Z"
 ---
 # Context And Session Workflow Spec
 
 Created: 2026-04-11T21:53:09Z
 
 ## Why
 
+Context and session enforcement should help an agent stay focused and leave the repo in a better state. Right now the repo contract is strong, but context loading is heavier than it needs to be and session closeout is stricter than it is helpful when memory updates are missing.
+
 ## Problem
 
+Brain's current context model is effectively all-or-nothing: agents either read a broad set of files or risk missing something important. Session finish can also detect missing durable memory work without giving the user a first-class recovery path. The result is friction where Brain should be guiding the next step.
+
 ## Goals
 
+- Introduce layered context loading so agents can start from a minimal contract and pull deeper context on demand.
+- Make session closeout point users toward the right recovery action when durable memory updates are missing.
+- Keep enforcement explicit and deterministic while reducing avoidable workflow friction.
+
 ## Non-Goals
 
+- Background automatic context loading with opaque heuristics.
+- Silent memory writes at session finish.
+- A durable long-term store for temporary scratch memory.
+
 ## Requirements
 
+- Add a context-loading interface that supports at least four levels: always-on identity/current-state, lightweight summaries, full context files, and deep search-driven context.
+- Make the active context tiering obvious enough that agents can request more depth intentionally.
+- When session finish sees meaningful repo change without memory updates, suggest `brain distill --session` or equivalent as the first recovery path.
+- Reuse the same distillation primitives as the planning-memory workflow instead of inventing a separate closeout-only mechanism.
+- Treat any session memory cache as optional, temporary, and explicitly non-durable until distilled.
+
 ## UX / Flows
 
+Layered context flow:
+1. Agent loads minimal startup context.
+2. Agent requests the next level only when the task needs more detail.
+3. Brain makes the transition from static context to search-driven deep context explicit.
+
+Session closeout recovery flow:
+1. User runs `brain session finish`.
+2. Brain detects repo changes without acceptable memory updates.
+3. Brain suggests a session-scoped distill flow instead of only returning a hard stop.
+4. User reviews the proposed memory updates, accepts what matters, and retries closeout.
+
 ## Data / Interfaces
 
+- Define context tiers in Brain-managed docs or config in a way that remains deterministic.
+- Thread active session information into distill so closeout can scope the candidate updates.
+- Keep any temporary session cache out of the accepted durable-memory globs unless explicitly promoted.
+
 ## Risks / Open Questions
 
+- How much of layered context belongs in generated docs versus command output?
+- Should session closeout suggest distill unconditionally, or only when certain note classes are missing?
+- Does a temporary session cache help enough to justify another concept in the workflow?
+
 ## Rollout
 
+1. Add layered context-loading primitives and documentation.
+2. Add session closeout suggestions that reuse distill once manual distill exists.
+3. Evaluate whether a temporary session cache is needed after those two steps land.
+
 ## Story Breakdown
 
+- [ ] Add Layered Context Loading Levels
+- [ ] Suggest Distillation During Session Closeout
+
 ## Resources
 
+- [[.brain/brainstorms/mempalace-inspired-brain-improvements.md]]
+- [[.brain/planning/stories/context-and-session-workflow-current-state-and-next-actions.md]]
+- [[.brain/resources/references/agent-workflow.md]]
+
 ## Notes
+
+The optional session-memory-cache idea stays out of the initial execution set until the lower-friction closeout path is proven useful.

.brain/planning/specs/planning-and-brainstorming-ux.md

@@ -2,35 +2,85 @@
 created: "2026-04-11T21:53:09Z"
 epic: planning-and-brainstorming-ux
 project: brain
-status: draft
+status: approved
 title: Planning And Brainstorming UX Spec
 type: spec
-updated: "2026-04-11T21:53:09Z"
+updated: "2026-04-13T06:23:52Z"
 ---
 # Planning And Brainstorming UX Spec
 
 Created: 2026-04-11T21:53:09Z
 
 ## Why
 
+Brain is strongest when durable project memory stays explicit, local, and reviewable. The main weakness in that model is that valuable reasoning often never becomes durable memory at all. This planning pass focuses on closing that gap without sliding into transcript hoarding.
+
 ## Problem
 
+Today Brain depends on explicit `brain edit` discipline and manual note updates. That preserves correctness, but it also means decisions, tradeoffs, bugs, and discoveries from active sessions can disappear unless someone remembers to capture them. The planning and brainstorming flow does not yet provide a first-class bridge from session work to curated project memory.
+
 ## Goals
 
+- Add an optional memory distillation workflow that turns session activity into proposed durable-note updates.
+- Preserve decision rationale as a first-class artifact instead of only capturing final outcomes.
+- Keep Brain's explicit approval model intact so distillation never becomes silent background storage.
+- Make the new memory-capture path feel native to the existing brainstorm -> epic -> spec -> story workflow.
+
 ## Non-Goals
 
+- Storing full conversations or command transcripts as durable memory by default.
+- Auto-writing AGENTS, context, or resource notes without review.
+- Replacing existing explicit note-edit workflows for users who prefer manual capture.
+
 ## Requirements
 
+- Add a `brain distill` workflow that can inspect recent session history, command runs, and git diff.
+- Distillation must output candidate updates for Brain-managed targets such as `AGENTS.md`, `.brain/context/*`, `.brain/resources/changes/*`, and decision notes.
+- Introduce a first-class decision note shape under `.brain/resources/decisions/` with enough structure to preserve context, options, decision, and tradeoffs.
+- Keep approval in the loop so users or agents can accept, reject, or edit generated updates before they become durable memory.
+- Record enough provenance that a later reader can tell which session or source material drove the distilled update.
+
 ## UX / Flows
 
+Manual distillation flow:
+1. User or agent runs `brain distill`.
+2. Brain gathers recent session history, command runs, and git diff context.
+3. Brain proposes note updates grouped by destination and reason.
+4. User or agent reviews, edits, and accepts the proposed updates.
+
+Decision-note flow:
+1. User or distill flow creates a decision note.
+2. The note captures context, options considered, final choice, and tradeoffs.
+3. The note is linked from specs, stories, or change notes where relevant.
+
 ## Data / Interfaces
 
+- Add a decision-note template and note-type conventions under `.brain/resources/decisions/`.
+- Define a distillation payload shape that can represent proposed file targets, generated body changes, and source provenance.
+- Keep generated output text-first and markdown-native so it fits the rest of Brain's note model.
+
 ## Risks / Open Questions
 
+- How much source material should distill inspect by default before the output becomes noisy?
+- Should approval happen inline in the terminal, through note creation, or via staged patch-like files?
+- How deterministic does the distilled proposal need to be for tests and history expectations?
+
 ## Rollout
 
+1. Add note templates and data structures for decision notes and distillation proposals.
+2. Ship a manual `brain distill` path first.
+3. Reuse the same primitives from session closeout once the manual workflow feels stable.
+
 ## Story Breakdown
 
+- [ ] Add Session-Aware Memory Distillation
+- [ ] Capture Decision Rationale As First-Class Notes
+
 ## Resources
 
+- [[.brain/brainstorms/mempalace-inspired-brain-improvements.md]]
+- [[.brain/planning/stories/planning-and-brainstorming-ux-current-state-and-next-actions.md]]
+
 ## Notes
+
+The guiding constraint is simple: recover missing insight without turning Brain into a raw transcript archive.