Writ coordinator v1.0.4

Crawls can now be read by AI, personas can record their own sign-in, and a crawl whose login silently failed is refused instead of harvesting login screens.

This release includes four schema migrations (0019–0022). They are applied automatically on start by the container entrypoint, so the upgrade is unchanged:

git pull && docker compose up -d --build

If you run the coordinator with a custom command: that bypasses entrypoint.sh, apply them yourself with alembic upgrade head.

Added

AI reader for crawls (executor=ai). Every fetched page can be read against a plain-language instruction and come back as structured records instead of raw markdown — for data with no clean CSS selector. It runs on your own provider, so nothing is metered or billed.

executor is independent of render_mode: an AI crawl of a JavaScript-rendered site is executor=ai + render_mode=browser, a bulk pass over static pages is executor=ai + render_mode=http. Failure is contained rather than silent — a page whose read fails keeps its markdown, and a crawl started with the AI reader on while no provider is configured is refused up front instead of fetching a whole site to produce exactly what a regular crawl would have.

Let AI sign a persona in, and keep the recording. A persona could previously only get a login workflow if you recorded one by hand or attached one you already had. The coordinator can now ask a connected fleet agent to sign in with the persona's credentials, record the flow, turn it into a workflow and wire it onto the persona — which is then replayed on every session expiry. The coordinator runs no AI of its own; the agent carries its own keys.

Changed

MCP run_<name> tools are now opt-in per workflow. The /mcp server used to mint one tool for every saved workflow. MCP clients inject every advertised tool schema into model context on each request and several enforce hard tool caps, so an instance with many workflows either burned thousands of tokens per turn or had its tool list truncated at the client. Pin the ones you want exposed; everything stays reachable through writ_run_workflow regardless. Existing workflows default to off.

The extracted-data endpoints serve from bounded reads. Every request used to load the full payload of the whole scan window and flatten it in Python just to answer "the newest 50 rows". On a crawl dataset that was tens of megabytes of JSON parsed per request, and the Data page fires several concurrently on open and then polls.

Fixed

A crawl could bank a site's sign-in page as content, 1069 times. Every check applied to a warm session inspected its shape — does it carry cookies, is one HttpOnly, does an auth-looking name appear. Plenty of sites hand an anonymous visitor a perfectly well-formed HttpOnly session cookie, so a login that silently failed produced a session that passed every test there was. The crawl then ran to completion capturing the logged-out page, and nothing in the pipeline could tell.

Sessions are now probed behaviourally — the site itself is asked whether the session is signed in — and a crawl whose persona turns out to be signed out is refused up front rather than producing a corpus of login screens.

Inline extraction returned page furniture instead of the article. The paths that extract without a fleet agent used a regex block-tag stripper whose "first N lines" are a page's nav and header. Those paths now mirror the agent's own extraction ladder server-side.

transfer_imports could fail to be created at all. created_by_user_id was declared String(36) while users.id is a UUID column, and the model placed a foreign key across that mismatch. A strict engine refuses such a constraint outright, so Base.metadata.create_all died on this table and took the rest of the schema with it.

Verification

Tagged only after scripts/release-e2e.sh passed against this commit: docker compose up → create the owner → install the published writ-agent release through the same one-liner the UI prints, checksum verified → record a workflow on a real browser → replay it → call it over REST and over MCP.

706 tests pass, all 22 migrations apply to a single head, and dependency audits are clean on both sides — npm audit 0, pip-audit 0, bandit clean.

You can run the same gate yourself with ./scripts/release-e2e.sh.

The tag is signed — git verify-tag v1.0.4.

Full changelog: CHANGELOG.md · v1.0.3…v1.0.4