Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 36 additions & 1 deletion docs/operator-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ Design home: `docs/superpowers/specs/2026-07-16-keel-broker-abstraction-design.m
## Pre-live checklist

Run through this **before arming the agent for live trading**, and re-check after any change to the
Coinbase account.
Coinbase account. Item 3 is not a one-time check: it recurs **weekly** for as long as the agent
runs.

### 1. ⛔ Disable interest / rewards on idle balances — **required**

Expand Down Expand Up @@ -54,6 +55,40 @@ A zakat estimate (~2.5% of holdings' market value per lunar year) is a **positiv
item 1's prohibition, and it is informational: keel reports, you decide and discharge it. Tracked at
KB §33.1. No pre-live action; noted here so the account-level obligation set is complete in one place.

### 3. Withdrawal-capability attestation — **weekly refresh** (rail 17)

**Why.** Rail 17 (§65.4 *qabd*) halts all BUY entries unless the withdrawal-capability attestation is
fresh, and "fresh" means a **7-day TTL** (`WITHDRAWAL_ATTESTATION_TTL_SEC`,
`keel/execution/executor.py`). An expired attestation reads as UNKNOWN, and rail 17 fails closed on
unknown — live DCA buys are vetoed. That is not hypothetical: on **2026-08-14** it vetoed the only
live DCA signal because the attestation had lapsed, and as of 2026-08-17 every deployment's
attestation was weeks stale, so rail 17 was halting entries on the **live** deployment (it is a
`LIVE_STATE` rail, skipped in paper, where a stale attestation matters only to the status display).
Each deployment carries its own attestation (they do not share a database), so the live one must be
refreshed for the rail and the paper ones to quiet their status lines.

Rail 17 fails closed by itself — what it cannot do is refresh its own input, and that input is
deliberately human (see the warning below). This entry is the cadence obligation the rail cannot
enforce, which is why it lives here rather than in `guards.py`.

**How to verify.** `keel status` prints the rail-17 line with days-to-expiry (`attested, expires in
3d`, `EXPIRED 12d ago`, or `never attested`) — staleness is visible there *before* it vetoes, not
only in the veto log. `keel withdrawals show` reads the same state with the age to one decimal.

**Cadence.** Re-attest **weekly** — a calendar reminder is the intended mechanism. Confirm the
balances really are withdrawable on demand, then, per deployment:

```bash
keel withdrawals attest --enabled
keel --config config.live-sandbox.yaml --db keel-live.db withdrawals attest --enabled
```

> ⚠️ **The typed confirmation is deliberately human.** `--enabled` RELEASES a rail-17 entry halt and
> demands a typed `yes` at a terminal — so that a scheduled job can never release a §65.4 halt, the
> same posture as `keel autonomy on`. Do not script this command and do not pipe a `yes` into it:
> the weekly habit is the fix for staleness, and automating the release would undo the rail. If a
> calendar reminder ever feels like it should be a cron job, re-read this warning.

---

## Adding to this file
Expand Down
12 changes: 7 additions & 5 deletions keel/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,19 @@
Wires the merged Phase 1-3 modules into a `click` CLI: `db import` (`data.csv_import.import_dir`),
`monitor` (`data.market_feed`), `agent` (`agent.run_once`/`agent.loop`), `autonomy on|off|show`
(the `profile` row `agent.run_once` itself re-reads each cycle),
`rules list|backtest|promote|demote|disable|seed|add` (`data.repository` + `strategy.backtest`/
`promotion`; `seed` populates the otherwise-empty `rules` table from `agent.RULE_REGISTRY`,
Issue #81, and `add` inserts ONE `candidate` row from operator-supplied `--params` JSON so a
proposed parameter set can reach `rules backtest` without hand-written Python),
`rules list|backtest|promote|demote|disable|enable|seed|add` (`data.repository` +
`strategy.backtest`/`promotion`; `seed` populates the otherwise-empty `rules` table from
`agent.RULE_REGISTRY`, Issue #81, and `add` inserts ONE `candidate` row from operator-supplied
`--params` JSON so a proposed parameter set can reach `rules backtest` without hand-written
Python),
`pnl` (`analysis.pnl`), `kill`/`resume` (the `agent_state` kill-switch),
`resume-entries` (clear an armed rail-16 consecutive-loss halt), `record-flow`
(declare a deposit/withdrawal so rail 11 does not read it as P&L) and `reset-hwm`
(reset rail 11's high-water mark),
`subscription attest|set|show` (the per-venue, user-attested allowance rail 14 reads live),
`status` (`commands.status`: the read-only, no-broker operator dashboard the paper-mode-fidelity
spec deferred -- mode/kill-switch/autonomy/Rail 11/positions/rules/data freshness, plus `--json`),
spec deferred -- mode/kill-switch/autonomy/Rail 11/rail 17 attestation freshness/positions/rules/
data freshness, plus `--json`),
and `insights summary|journal` (`commands.insights`: a read-only VIEW over the same substrate --
per-rule promotion-gate distance and a filterable trade journal, also with `--json`).

Expand Down
43 changes: 42 additions & 1 deletion keel/commands/rules.py
Original file line number Diff line number Diff line change
Expand Up @@ -402,13 +402,54 @@ def rules_demote(ctx: click.Context, rule_id: int) -> None:
@click.pass_context
@with_disclaimer
def rules_disable(ctx: click.Context, rule_id: int) -> None:
"""Disable a rule (terminal status; it will never trade again)."""
"""Disable a rule (nothing promotes from here; `keel rules enable` can restore it, at
`candidate` -- never at the status it held when disabled)."""
repo = _open_repo(ctx)
row = _require_rule_row(ctx, repo, rule_id)
repo.update_rule_status(rule_id, "disabled")
click.echo(f"rule {rule_id} ({row['kind']}): status -> disabled")


@rules_group.command("enable")
@click.argument("rule_id", type=int)
@click.pass_context
@with_disclaimer
def rules_enable(ctx: click.Context, rule_id: int) -> None:
"""Re-enable a disabled rule at `candidate` -- the bottom of the lifecycle ladder.

This is the inverse of `rules disable`'s WRITE, not of its effect: `disable` records
nothing about the status a rule held before it was disabled (it stamps only `demoted_at`),
so there is nothing to restore, and a rule disabled from `live` comes back as `candidate`
too. That is deliberate -- re-entry to the trading set is a promotion decision the gate
must see again, not an undo.

The path onward from `candidate` is `keel rules promote <id>` (gated); its `--force` is
the documented bypass for a paper-forward whose backtest can never reach the min_trades
floor -- a DCA rule produces no backtest trades at all, so force is the only way it can
reach `paper` (see `promote`'s own docstring).
"""
repo = _open_repo(ctx)
row = _require_rule_row(ctx, repo, rule_id)
if row["status"] != "disabled":
click.echo(
f"Error: rule {rule_id} ({row['kind']}) is {row['status']!r}, not disabled -- "
"`enable` only restores a disabled rule. To advance this one, use "
"`keel rules promote`.",
err=True,
)
ctx.exit(1)
return
repo.update_rule_status(rule_id, "candidate")
click.echo(f"rule {rule_id} ({row['kind']}): status -> candidate")
click.echo(
f"rule {rule_id} ({row['kind']}): re-enabled at CANDIDATE, the lifecycle floor -- "
"a rule disabled from `live` lands here too (disable records no prior status to "
f"restore). Advance with `keel rules promote {rule_id}`; `--force` is the documented "
"bypass for a paper-forward whose backtest can never reach the min_trades floor "
"(e.g. DCA)."
)


def _json_plain(value: Any) -> Any:
"""Coerce `value` into the JSON-plain form `Repository.insert_rule` expects for `params`.

Expand Down
124 changes: 118 additions & 6 deletions keel/commands/status.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,9 @@
("A dedicated `keel status` command is deferred as a follow-up"); this is that follow-up. Its
job is narrow: let an operator running a paper-forward see the agent's state at a glance,
**purely from the local DB and config** -- mode, kill-switch, autonomy, Rail 11 drawdown/equity
state, open positions, rule counts, and per-product data freshness. It NEVER calls the broker;
that is the whole point (`monitor`/`agent` are the commands that touch the network).
state, rail 17's withdrawal-attestation freshness, open positions, rule counts, and per-product
data freshness. It NEVER calls the broker; that is the whole point (`monitor`/`agent` are the
commands that touch the network).

Two layers, matching the rest of `keel/commands/*`:

Expand All @@ -32,6 +33,7 @@
from keel.commands._products import _default_sim_products
from keel.config import Config
from keel.data.repository import Repository
from keel.execution import executor
from keel.types import Granularity

# -- the pure report shape ---------------------------------------------------------------------
Expand Down Expand Up @@ -86,6 +88,24 @@ class SubscriptionStatusRow:
effective_cap: Decimal | None


@dataclass(frozen=True)
class WithdrawalAttestationStatus:
"""Rail 17's input, as a displayable state (§65.4) -- the same reading `keel withdrawals
show` prints, so the two commands can never disagree about whether entries are halted.

`expires_in_sec` is set only when `state == "attested"` (a fresh `--enabled` attestation);
`expired_for_sec` only when `state == "expired"` -- how long ago the 7-day TTL lapsed, not
how old the attestation is, because "unknown for 12 days" is the fact that matters to an
operator deciding how urgently to re-attest.
"""

state: str # "attested" | "suspended" | "expired" | "unattested"
enabled: bool | None
attested_at: int | None
expires_in_sec: int | None
expired_for_sec: int | None


@dataclass(frozen=True)
class StatusReport:
now_ts: int
Expand All @@ -99,6 +119,7 @@ class StatusReport:
max_total_dd_pct: Decimal
max_weekly_dd_pct: Decimal
rail11_status: str
withdrawal_attestation: WithdrawalAttestationStatus
paper_cash_usdc: Decimal | None
open_positions: list[OpenPositionStatus]
rule_counts: dict[str, int]
Expand Down Expand Up @@ -211,6 +232,47 @@ def _subscription_rows(
]


def _withdrawal_attestation(repo: Repository, now_ts: int) -> WithdrawalAttestationStatus:
"""Rail 17's input, resolved and aged for display.

Resolution goes through the executor's OWN `_withdrawals_enabled` and expiry math goes
through the executor's OWN `WITHDRAWAL_ATTESTATION_TTL_SEC` -- never a restated 7 days --
so this display can never call an attestation fresh on the very cycle the rail vetoes it.
Staleness takes precedence over suspension, matching `withdrawals show`: a stale attested
suspension is UNKNOWN, not a fresh claim that withdrawals are down.

The corrupt-state read is guarded the way the executor guards its own (`try/except` ->
unknown): a dashboard must not inherit a crash path the rail deliberately does not have.
"""
resolved = executor._withdrawals_enabled(repo, now_ts)
try:
attested_at = int(repo.get_state("withdrawals_attested_at", default=0) or 0)
except (TypeError, ValueError):
attested_at = 0

if not attested_at:
return WithdrawalAttestationStatus("unattested", None, None, None, None)
enabled_flag = repo.get_state("withdrawals_enabled", default=None)
if resolved is None and enabled_flag is None:
# `withdrawals show`'s UNKNOWN corner, kept word-for-word: an attested_at exists but
# the enabled flag itself is unreadable/absent, which only DB surgery produces (the
# CLI writes both keys). Not "expired" -- the staleness question was never reached.
return WithdrawalAttestationStatus("unknown", None, attested_at, None, None)
if resolved is None:
expired_for = max((now_ts - attested_at) - executor.WITHDRAWAL_ATTESTATION_TTL_SEC, 0)
return WithdrawalAttestationStatus(
"expired",
enabled_flag,
attested_at,
None,
expired_for,
)
if resolved is False:
return WithdrawalAttestationStatus("suspended", False, attested_at, None, None)
expires_in = max((attested_at + executor.WITHDRAWAL_ATTESTATION_TTL_SEC) - now_ts, 0)
return WithdrawalAttestationStatus("attested", True, attested_at, expires_in, None)


def gather_status(repo: Repository, config: Config, now_ts: int) -> StatusReport:
"""Assemble the full status report from `repo`/`config` alone -- no broker, no network.

Expand Down Expand Up @@ -241,6 +303,7 @@ def gather_status(repo: Repository, config: Config, now_ts: int) -> StatusReport
max_total_dd_pct=max_total,
max_weekly_dd_pct=max_weekly,
rail11_status=_rail11_status(dd_total, dd_weekly, max_total, max_weekly),
withdrawal_attestation=_withdrawal_attestation(repo, now_ts),
paper_cash_usdc=(
repo.get_state("paper_cash_usdc") if config.auto_trade.mode == "paper" else None
),
Expand Down Expand Up @@ -268,6 +331,52 @@ def _human_age(age_sec: int) -> str:
return f"{days}d ago"


def _human_remaining(remaining_sec: int) -> str:
"""`_human_age`'s mirror image, for time still left rather than time already gone -- the
same ladder (s/m/h/d) so the two read as one convention on adjacent lines."""
if remaining_sec < 60:
return f"{remaining_sec}s"
minutes = remaining_sec // 60
if minutes < 60:
return f"{minutes}m"
hours = remaining_sec // 3600
if hours < 24:
return f"{hours}h"
days = remaining_sec // 86400
return f"{days}d"


def _rail17_line(w: WithdrawalAttestationStatus, rail_evaluated: bool) -> str:
"""The rail-17 line, naming the halt and the release in the same breath whenever entries
are halted -- the 2026-08-14 event was invisible until the veto fired precisely because
nothing surfaced the attestation's age BEFORE rail 17 acted on it (#340).

`rail_evaluated` is False in paper mode: rail 17 is a LIVE_STATE rail, skipped offline,
so a stale attestation halts nothing there. Claiming "entries halted" on a paper
dashboard would be a permanently-red alert for a halt that cannot occur -- the exact
alert-fatigue failure #340 exists to fix -- so the paper rendering names the state and
says the rail is not evaluated, and the re-attest prompt is kept (the LIVE deployment's
attestation is refreshed by the same typed command, and paper status should still nudge).
"""
prefix = "rail 17 (withdrawal capability)"
if w.state == "attested":
return f"{prefix}: attested, expires in {_human_remaining(w.expires_in_sec or 0)}"
if w.state == "unknown":
return f"{prefix}: UNKNOWN (state unreadable); re-attest with keel withdrawals attest"
halt = " -- entries halted" if rail_evaluated else " (rail 17 not evaluated in paper)"
if w.state == "suspended":
return (
f"{prefix}: SUSPENDED{halt}; "
"re-attest with keel withdrawals attest --enabled"
)
if w.state == "expired":
return (
f"{prefix}: EXPIRED {_human_age(w.expired_for_sec or 0)}{halt}; "
"re-attest with keel withdrawals attest"
)
return f"{prefix}: never attested{halt}; re-attest with keel withdrawals attest"


def render_human(report: StatusReport) -> list[str]:
"""The `keel status` (default, non-`--json`) rendering, as a list of lines -- kept as a pure
function of the report so it is testable without a CliRunner."""
Expand Down Expand Up @@ -299,6 +408,7 @@ def render_human(report: StatusReport) -> list[str]:
f"weekly={dd_weekly} (ceiling {report.max_weekly_dd_pct})"
)
lines.append(f"rail11 (drawdown breaker): {report.rail11_status}")
lines.append(_rail17_line(report.withdrawal_attestation, report.mode != "paper"))
if report.mode == "paper":
lines.append(f"paper_cash_usdc: {report.paper_cash_usdc}")

Expand Down Expand Up @@ -355,10 +465,12 @@ def _report_to_jsonable(report: StatusReport) -> dict[str, Any]:
def status_cmd(ctx: click.Context, as_json: bool) -> None:
"""Operator-observability snapshot of the agent's state -- read-only, no broker call.

Shows mode, the kill-switch, autonomy, Rail 11 drawdown/equity state, open positions, rule
counts, and per-product data freshness, all read straight from the local DB and config. This
is the interim of the deferred `keel status` TUI: same underlying report (`--json` is its
forward-compatible shape), just rendered to the terminal.
Shows mode, the kill-switch, autonomy, Rail 11 drawdown/equity state, rail 17's
withdrawal-attestation state (fresh with time remaining, or EXPIRED with how long ago and
the command that re-attests), open positions, rule counts, and per-product data freshness,
all read straight from the local DB and config. This is the interim of the deferred
`keel status` TUI: same underlying report (`--json` is its forward-compatible shape), just
rendered to the terminal.

`--json` deliberately skips the disclaimer footer every other command prints
(`with_disclaimer`): it exists for scripting/the future TUI, and a trailing line of prose
Expand Down
13 changes: 12 additions & 1 deletion keel/commands/tui.py
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@
render_propose_view,
render_screen_report,
)
from keel.commands.status import StatusReport, _human_age, gather_status
from keel.commands.status import StatusReport, _human_age, _rail17_line, gather_status
from keel.config import Config
from keel.data.repository import Repository
from keel.types import Granularity
Expand Down Expand Up @@ -288,6 +288,17 @@ def _equity_lines(report: StatusReport) -> list[ScreenLine]:
)
rail11_text = f"rail11 (drawdown breaker): {report.rail11_status}"
lines.append(ScreenLine(rail11_text, _rail11_style(report.rail11_status)))
# `render_human`'s exact rail-17 text, so the TUI and `keel status` can never disagree
# about whether entries are halted. Every state but `attested` fails rail 17 closed --
# a halt -- so each is an alert; EXCEPT in paper mode, where the rail is not evaluated
# and a stale attestation halts nothing (a permanently-red alert there is fatigue, not
# information), so the same states downgrade to warn.
rail17 = report.withdrawal_attestation
rail17_evaluated = report.mode != "paper"
style = "ok"
if rail17.state != "attested":
style = "alert" if rail17_evaluated else "warn"
lines.append(ScreenLine(_rail17_line(rail17, rail17_evaluated), style))
if report.mode == "paper":
lines.append(ScreenLine(f"paper_cash_usdc: {report.paper_cash_usdc}", "normal"))
return lines
Expand Down
5 changes: 3 additions & 2 deletions keel/strategy/promotion.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,9 @@
from keel.strategy.backtest import BacktestResult

# Lifecycle order; `transition()` advances one step at a time (or demotes to the
# terminal state). `disabled` is terminal in v1 — reactivation is a future decision,
# not modeled here.
# terminal state). `disabled` is terminal ON THIS PATH — reactivation is the operator
# verb `keel rules enable` (keel/commands/rules.py), which restores the row at
# `candidate`, deliberately not through the promotion machinery.
_PROMOTE_NEXT: dict[str, str] = {"candidate": "paper", "paper": "live"}


Expand Down
6 changes: 5 additions & 1 deletion tests/commands/test_insights.py
Original file line number Diff line number Diff line change
Expand Up @@ -322,7 +322,7 @@ def test_rule_track_record_degrades_gracefully_for_unrecognized_kind(repo: Repos


def test_account_summary_projects_verbatim_from_status_report() -> None:
from keel.commands.status import AutonomyStatus
from keel.commands.status import AutonomyStatus, WithdrawalAttestationStatus

status_report = StatusReport(
now_ts=NOW_TS,
Expand All @@ -339,6 +339,10 @@ def test_account_summary_projects_verbatim_from_status_report() -> None:
max_total_dd_pct=Decimal("0.20"),
max_weekly_dd_pct=Decimal("0.08"),
rail11_status="ok",
withdrawal_attestation=WithdrawalAttestationStatus(
state="attested", enabled=True, attested_at=NOW_TS - 86400,
expires_in_sec=6 * 86400, expired_for_sec=None,
),
paper_cash_usdc=Decimal("955.25"),
open_positions=[],
rule_counts={},
Expand Down
Loading
Loading