Skip to content

feat(#39): stable surface — SQLite queue, extras gate, drop redis/cluster#42

Merged
hadamrd merged 1 commit into
trunkfrom
loop/39-epic-ship-a-stable-surface-quarantine-ex
May 27, 2026
Merged

feat(#39): stable surface — SQLite queue, extras gate, drop redis/cluster#42
hadamrd merged 1 commit into
trunkfrom
loop/39-epic-ship-a-stable-surface-quarantine-ex

Conversation

@hadamrd
Copy link
Copy Markdown
Owner

@hadamrd hadamrd commented May 27, 2026

Closes #39.

Summary

Quarantine experimental features behind pip install 'forge-loop[experimental]',
swap the Redis-backed queue + leader-election cluster code for a durable
SQLite-WAL queue, and document the stability matrix.

The product is one operator on one box; the default install now reflects
that. Everything else is opt-in.

Acceptance criteria → evidence

  1. Stability matrix in README
    README.md now opens with a STABLE / EXPERIMENTAL / REMOVED table
    that maps every module to its tier.
  2. pyproject.toml extras
    Default deps shrunk to: mcp, pyyaml, duckdb, anyio, anthropic,
    claude-agent-sdk. New [experimental] extra carries fastapi,
    uvicorn, prometheus_client, opentelemetry-*, jinja2. The redis +
    observability extras are gone. Each experimental module top-imports
    forge_loop._extras.require_experimental(...) — a clear ImportError
    names the extra to install if not present (override via
    FORGE_LOOP_EXPERIMENTAL=1 for local dev).
  3. SQLite queue replaces Redis
    • forge_loop.queue.sqlite.SQLiteQueue — WAL mode, FIFO via
      enqueued_at, ack/nack with head-bias on requeue, lock-guarded
      for concurrent producers. New production default.
    • InMemoryQueue stays as the test default.
    • forge_loop.queue.redis_backend and forge_loop.cluster.*
      deleted; build_queue('redis://...') raises a clear ValueError
      pointing to SQLite.
  4. Critic prompt patch
    critic.md.tmpl now lists product as a category and teaches the
    no-scaffold-theatre rule (sev2 finding for orphan backends /
    adapters). critic.VALID_CATEGORY accepts the new category.
  5. README "running on a Claude subscription"
    Spelled out: per-token budget tracking is unsupported in flat-fee
    mode; wall-clock + cooldowns are the safety nets.

Test matrix

  • tests/test_queue_sqlite.py — happy-path FIFO, ack/nack semantics,
    timeout returns None, durability across reopen, attempts counter,
    concurrent multi-thread pop with no double-delivery, factory
    routing for sqlite/memory/redis URLs.
  • tests/test_install_surface.py — every stable module imports clean;
    every experimental module raises ImportError when the gate is
    forced off; removed modules stay removed (regression guard); runner
    source contains no ClusterCoordinator / redis_backend / cluster
    import; critic accepts product; pyproject defaults contain none of
    fastapi/uvicorn/prometheus_client/redis/jinja2.
  • tests/conftest.py — sets FORGE_LOOP_EXPERIMENTAL=1 so the
    existing experimental-module test files keep importing; the
    install-surface tests clear it via monkeypatch when simulating a
    default install (save/restore sys.modules so they don't pollute
    later tests' class identities).

Full suite: 413 passed, 1 skipped (skip is the live SDK integration
that needs an API key). ruff check src/ tests/ clean.

Test plan

  • uv run pytest tests/test_queue_sqlite.py tests/test_install_surface.py
  • uv run pytest tests/ (full suite stays green)
  • uv run ruff check src/ tests/

🤖 Generated with Claude Code

…ster

Quarantine experimental features behind the [experimental] extra and
shrink the default install surface to what a single operator actually
runs. Replace the Redis-backed queue + leader-election cluster code
with a durable SQLite queue (WAL mode) — zero infra, ACID, survives
crashes, and matches the supported deployment shape (one operator on
one box).

Why this change exists
----------------------
After the 16-PR self-bootstrap sprint we shipped ~9.5k LOC but only
~30% of it has a real downstream consumer. The rest is scaffolding:
Redis queue, leader election, multirepo, async runner, slack/discord/
webhook adapters, dashboard, prometheus/otel exporters, replay,
pipeline DAG. The product is one operator on one box, so the default
surface should match — and the scaffolding should be opt-in instead
of pulling third-party deps into every install.

What changed
------------
* New `forge_loop._extras.require_experimental()` — each experimental
  module calls it at import time; raises a clear `ImportError` naming
  the extra to install if not present. `FORGE_LOOP_EXPERIMENTAL=1`
  bypasses the gate for local dev.
* New `forge_loop.queue.sqlite.SQLiteQueue` — durable embedded queue
  with WAL journaling, FIFO order, ack/nack semantics, concurrent-safe
  via a lock. Becomes the production-grade default; in-memory stays
  the test default.
* `pyproject.toml`: default install drops the redis + observability
  extras and ships only the stable deps. New `[experimental]` extra
  pulls fastapi, uvicorn, prometheus_client, opentelemetry-*, jinja2.
* Deleted: `forge_loop.queue.redis_backend`, `forge_loop.cluster.*`,
  `tests/test_queue.py` (redis-fakeredis tests), `tests/test_cluster.py`.
  `build_queue('redis://...')` now raises a clear ValueError pointing
  at SQLite.
* Runner: drop ClusterCoordinator; wire the queue directly.
* CLI: `cluster status` becomes a deprecated stub returning a clear
  error; `--queue` help text now mentions sqlite.
* Critic brief + `VALID_CATEGORY`: add `product` category and the
  no-scaffold-theatre rule — orphan backends/adapters now earn a sev2
  product finding.
* README: stability matrix marking each module STABLE / EXPERIMENTAL /
  REMOVED; "Running on a Claude subscription" section spelling out
  that per-token budget tracking is unsupported in flat-fee mode.

Tests
-----
* `tests/test_queue_sqlite.py` — happy-path FIFO, ack/nack, durability
  across reopen, concurrent-pop adversarial check, factory routing.
* `tests/test_install_surface.py` — stable modules import clean,
  experimental modules refuse without the extra (with sys.modules
  save/restore so we don't pollute later tests), removed modules stay
  removed, runner has no cluster import, critic accepts `product`,
  pyproject defaults are minimal.
* `tests/conftest.py` — sets `FORGE_LOOP_EXPERIMENTAL=1` so existing
  experimental-module tests keep working in the dev environment;
  install-surface tests clear it via monkeypatch when simulating a
  default install.

Closes #39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

EPIC: ship a 'stable surface' — quarantine experimental features into [extras], swap Redis→SQLite, document the matrix

1 participant