Skip to content

v3.3.0 — Experiment observability, benchmark authoring, and human review

Choose a tag to compare

View all tags
@drawal1 drawal1 released this 10 Sep 17:50
· 7 commits to main since this release
v3.3.0
d9f0fe5
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

v3.3.0

Adds a complete experiment and review workflow on top of the observability store: versioned benchmarks and tasks, registered experiments, execution evidence, and human feedback anchored to the turn or component that needs work (#82). Package version is 3.3.0 (pyproject.toml). This wheel is already on PyPI.

There were no GitHub tags for v3.2.0 or v3.2.1 (those versions did ship on PyPI). This tag is everything since v3.1.2.

This is not a drop-in import-path or store-file upgrade. Read the breaking section before you open an existing observability.sqlite3 or keep scripts that import the old module names.

Requires Python >=3.13,<3.15 (unchanged). Restart run_chatbot to load the updated UI.

Breaking changes (read before upgrading)

  1. Admin dump/token routes stay off by default (v3.2.0).
    • POST /admin/dump_all_conversations and POST /admin/generate_mcp_token 404 unless the server is started with --enable_admin_endpoints, and they need a Bearer token even then.

Routing-adjacent behavior

Nothing here changes the default install's routing unless you set the new env knobs.

  1. Tied fuzzy command-name pre-matches still defer to the classifier (shipped in v3.1.2). Prefix-sharing commands no longer pick [0] during INTENT_DETECTION. Clarification stages are unchanged.

  2. Optional experiment isolation knobs (defaults leave today's behavior):

    • FW_LM_CACHE=0 — disable the process-wide DSPy LM cache so repeated attempts are not byte-identical cache hits.
    • FW_UTTERANCE_CACHE_SCOPE=workflow — shard the utterance/clarification cache by workflow id so attempts do not inherit each other's disambiguations.
    • The experiment runner applies these for measured runs. Unset keeps the shared cache and DSPy cache as before.

Highlights

  • Experiment container — driver-neutral lifecycle, exact attempt declarations, bootstrap/claim binding and fencing, evidence draining, sealed workspace browsing. Creation mints stable IDs for runner handoff; it does not execute tasks.
  • Observability store and run_chatbot UI — one observability.sqlite3 per workflow (conversations, turns, OTel-shaped spans, artifacts, train runs, writer health). Debug sidebar: benchmark → experiment → conversation; ad-hoc conversations grouped by UTC date. Turns and trace components open in the right pane with a full breadcrumb. Record navigator for moving between records without losing place.
  • Benchmarks — versioned catalog and setup in the chatbot and in fastworkflow.benchmark.
  • Human feedback and review sidecar — component-anchored, timestamped comments, kept separate from agent-memory feedback. Independent rubric-based review (observability-reviews.sqlite3): capability-gated assignments, blinded evidence projections, rating export. Feedback is editable in working stores and read-only in workspace snapshots.
  • Capture policy and decision signals — debug vs evidence profiles (FW_OBS_CAPTURE_PROFILE; default debug is today's verbatim capture plus credential scrub). Decision uncertainty/consequence are recorded; they do not change routing.
  • Archive experiments — durable archived annotation (schema v6); UI marks archived rows.
  • Runtime manifests and readiness — snapshots on bound attempts, structured execution records, provenance, cost visibility, writer-health improvements.
  • Developer skills for benchmark authoring, trace diagnosis, and turning feedback into regression coverage.
  • Training (v3.2.1 / #78) — thin (below-floor) intent labels warn and stay unmeasured instead of aborting train or blocking publish; empty/missing labels still block.

What's Changed (PRs since v3.1.2)

  • feat: v3.2.0 — observability store, chatbot debug UI, and conversation-store consolidation by @dharrawal in #73
  • Fix training behavior for thin labels and missing provenance by @dharrawal in #75
  • fix(train): do not refuse publication for below-floor labels by @dharrawal in #78
  • Add experiment observability, benchmark authoring and human review by @dharrawal in #82

Full Changelog: v3.1.2...v3.3.0

Install

uv pip install 'fastworkflow==3.3.0'
# optional extras:
uv pip install 'fastworkflow[server]==3.3.0' # FastAPI/MCP HTTP service
uv pip install 'fastworkflow[training]==3.3.0' # HuggingFace datasets for train

GPU training (optional — default install is CPU torch):

uv pip install --upgrade --force-reinstall 'torch>=2.7.1' \
 --index-url https://download.pytorch.org/whl/cu130
uv pip install --upgrade 'fsspec[http]>=2023.1.0,<=2026.2.0'
python -c "import torch; print(torch.cuda.is_available(), torch.__version__)"

Contributors

dharrawal
Assets 2
Loading