feat(#39): stable surface — SQLite queue, extras gate, drop redis/cluster#42
Merged
Merged
Conversation
…ster
Quarantine experimental features behind the [experimental] extra and
shrink the default install surface to what a single operator actually
runs. Replace the Redis-backed queue + leader-election cluster code
with a durable SQLite queue (WAL mode) — zero infra, ACID, survives
crashes, and matches the supported deployment shape (one operator on
one box).
Why this change exists
----------------------
After the 16-PR self-bootstrap sprint we shipped ~9.5k LOC but only
~30% of it has a real downstream consumer. The rest is scaffolding:
Redis queue, leader election, multirepo, async runner, slack/discord/
webhook adapters, dashboard, prometheus/otel exporters, replay,
pipeline DAG. The product is one operator on one box, so the default
surface should match — and the scaffolding should be opt-in instead
of pulling third-party deps into every install.
What changed
------------
* New `forge_loop._extras.require_experimental()` — each experimental
module calls it at import time; raises a clear `ImportError` naming
the extra to install if not present. `FORGE_LOOP_EXPERIMENTAL=1`
bypasses the gate for local dev.
* New `forge_loop.queue.sqlite.SQLiteQueue` — durable embedded queue
with WAL journaling, FIFO order, ack/nack semantics, concurrent-safe
via a lock. Becomes the production-grade default; in-memory stays
the test default.
* `pyproject.toml`: default install drops the redis + observability
extras and ships only the stable deps. New `[experimental]` extra
pulls fastapi, uvicorn, prometheus_client, opentelemetry-*, jinja2.
* Deleted: `forge_loop.queue.redis_backend`, `forge_loop.cluster.*`,
`tests/test_queue.py` (redis-fakeredis tests), `tests/test_cluster.py`.
`build_queue('redis://...')` now raises a clear ValueError pointing
at SQLite.
* Runner: drop ClusterCoordinator; wire the queue directly.
* CLI: `cluster status` becomes a deprecated stub returning a clear
error; `--queue` help text now mentions sqlite.
* Critic brief + `VALID_CATEGORY`: add `product` category and the
no-scaffold-theatre rule — orphan backends/adapters now earn a sev2
product finding.
* README: stability matrix marking each module STABLE / EXPERIMENTAL /
REMOVED; "Running on a Claude subscription" section spelling out
that per-token budget tracking is unsupported in flat-fee mode.
Tests
-----
* `tests/test_queue_sqlite.py` — happy-path FIFO, ack/nack, durability
across reopen, concurrent-pop adversarial check, factory routing.
* `tests/test_install_surface.py` — stable modules import clean,
experimental modules refuse without the extra (with sys.modules
save/restore so we don't pollute later tests), removed modules stay
removed, runner has no cluster import, critic accepts `product`,
pyproject defaults are minimal.
* `tests/conftest.py` — sets `FORGE_LOOP_EXPERIMENTAL=1` so existing
experimental-module tests keep working in the dev environment;
install-surface tests clear it via monkeypatch when simulating a
default install.
Closes #39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #39.
Summary
Quarantine experimental features behind
pip install 'forge-loop[experimental]',swap the Redis-backed queue + leader-election cluster code for a durable
SQLite-WAL queue, and document the stability matrix.
The product is one operator on one box; the default install now reflects
that. Everything else is opt-in.
Acceptance criteria → evidence
README.mdnow opens with a STABLE / EXPERIMENTAL / REMOVED tablethat maps every module to its tier.
pyproject.tomlextras ✅Default deps shrunk to: mcp, pyyaml, duckdb, anyio, anthropic,
claude-agent-sdk. New
[experimental]extra carries fastapi,uvicorn, prometheus_client, opentelemetry-*, jinja2. The redis +
observability extras are gone. Each experimental module top-imports
forge_loop._extras.require_experimental(...)— a clearImportErrornames the extra to install if not present (override via
FORGE_LOOP_EXPERIMENTAL=1for local dev).forge_loop.queue.sqlite.SQLiteQueue— WAL mode, FIFO viaenqueued_at, ack/nack with head-bias on requeue, lock-guardedfor concurrent producers. New production default.
InMemoryQueuestays as the test default.forge_loop.queue.redis_backendandforge_loop.cluster.*deleted;
build_queue('redis://...')raises a clearValueErrorpointing to SQLite.
critic.md.tmplnow listsproductas a category and teaches theno-scaffold-theatre rule (sev2 finding for orphan backends /
adapters).
critic.VALID_CATEGORYaccepts the new category.Spelled out: per-token budget tracking is unsupported in flat-fee
mode; wall-clock + cooldowns are the safety nets.
Test matrix
tests/test_queue_sqlite.py— happy-path FIFO, ack/nack semantics,timeout returns None, durability across reopen, attempts counter,
concurrent multi-thread pop with no double-delivery, factory
routing for sqlite/memory/redis URLs.
tests/test_install_surface.py— every stable module imports clean;every experimental module raises
ImportErrorwhen the gate isforced off; removed modules stay removed (regression guard); runner
source contains no
ClusterCoordinator/redis_backend/ clusterimport; critic accepts
product; pyproject defaults contain none offastapi/uvicorn/prometheus_client/redis/jinja2.
tests/conftest.py— setsFORGE_LOOP_EXPERIMENTAL=1so theexisting experimental-module test files keep importing; the
install-surface tests clear it via
monkeypatchwhen simulating adefault install (save/restore
sys.modulesso they don't pollutelater tests' class identities).
Full suite: 413 passed, 1 skipped (skip is the live SDK integration
that needs an API key).
ruff check src/ tests/clean.Test plan
uv run pytest tests/test_queue_sqlite.py tests/test_install_surface.pyuv run pytest tests/(full suite stays green)uv run ruff check src/ tests/🤖 Generated with Claude Code