Skip to content

Part 3: Self-improve, with receipts — the job agent that measures its own fixes

Choose a tag to compare

View all tags
@jamwithai jamwithai released this 13 Aug 10:47
· 71 commits to main since this release
part3.0
674802e

The Observable Job Agent, Part 3: Self-improve, with receipts.

An AI assistant in the loop made the search fifteen times faster, found a regression we had shipped a week earlier and never noticed, and got one thing confidently wrong. Our own gate looked at all of it and said you have not fixed it, and it was right.

📖 Blog post: https://jamwithai.substack.com/p/build-your-own-job-agent-part-3

What's inside

  • Per-source spans (traced_call): every job source is timed separately, which is what turned "search takes 15s" into "jsearch takes 15s and contributes nothing"
  • A two-phase soft deadline in the source cascade: bound the wait on the primary, fall through to the finished sources, then go back for the primary if the cascade is still short. 15.3s → 1.0s median, and the spread collapses from ~5s to 7ms
  • job-scout-search-suite: a regression gate that grades the cascade rather than the prose, written BEFORE the fix so it could still disagree afterwards. It did: 33% → 33%, because containing a slow source is not the same as making it fast
  • The HRPO-optimized tailor prompt, tuned against our own deterministic validator rather than an LLM judge: fabrication 0.2768 → 0.1288 on fresh live jobs
  • gates/, scripts/bench_search.py, the phase-3 measurement records, and docs/ollie.md: the full assistant transcript with what to check each answer against

Get started

git clone --branch part3.0 https://github.com/jamwithai/observable-job-agent
cd observable-job-agent
uv sync --all-groups
cp .env.example .env    # add one LLM key (OPENAI_API_KEY, or free via groq:/ollama:)
make test               # 129 tests, no network or keys needed
make gates              # the deterministic regression gate, no LLM calls
make app                # http://localhost:7860
Assets 2
Loading