📚 Crawls that keep the whole page — and self-hosted deploys that connect

Released 2026-09-23

A knowledge source that looked fine could be storing almost nothing. The extractor kept the first block of a page that cleared a length floor, and a hero banner clears a length floor — so eight of eighty-eight production sources were holding under 500 characters of the pages they had crawled. Nine organizations hit the related ingestion failure and all nine left. That is the main thing this release fixes.

Alongside it: plan limits that ignored crawls still in flight, the crawl surface behind a run of Search Console 404s, error responses that echoed our own exception text back to callers, and two fixes that matter only to self-hosters — a dashboard that hung at "Connecting to your agent…" behind HTTPS, and a console error on every page load.

Highlights

Crawls that keep the page, not the first banner

The extractor returned the first container whose text passed min_content_length. That floor was never a quality bar. eazzyliving.co.uk's first <article> is a 108-character search widget, so the widget was indexed and 4,500 characters of page were dropped; that customer retried four times and gave up. secondmile.org crawled ten pages and kept a 197-character hero banner from each.

Selection now takes the deepest container holding at least 90% of the page's non-boilerplate text. The walk lives in one place, shared with the help-center importer, which had been carrying its own copy.

A page that is nothing but navigation chrome now reports empty rather than handing that chrome back as content — which is how one site came to store "Welcome, Guest - Login" instead of its catalogue.

Measured against 27 live production sources: 12 improved, by 1.4x to 43x; 10 came back byte-identical; 3 now correctly fall through to the browser renderer. Of the ten domains that had failed outright, five now succeed. The other five are four domains with no A record and one host that drops the connection on both 80 and 443 — nothing can crawl those, and they now say which of those it is instead of guessing at three causes in one string.

Three more failures came out of the same work. The empty-crawl guard counted pages crawled rather than documents actually embedded, so a source whose pages crawled and then failed to embed was reported as a success with an empty vector table. Two sources queued together by one organization raced on CREATE TABLE and the loser died, losing that source. And hosts that were never reachable were retried three times plus a browser launch — 167 seconds across five dead domains, now 39.

Knowledge source limits count what is still in flight

A crawl took its plan slot only when it finished indexing, so every request in a batch read the same count and passed. A one-source plan accepted three sites in one sitting. Queued and running rows are counted now, at all three enforcement points. A row left processing by a restart stops holding a slot after six hours.

Locked features can open their upgrade dialog

Locked tabs and the Workflow toggle carried disabled, so the click never reached the handler and the upgrade dialog was simply unreachable. They stay dimmed and keep the lock icon. The dialog's overlay was also taller than a laptop viewport with no way to scroll, which put its own Upgrade button off-screen.

Closing the crawl surface behind the Search Console 404s

Search engines were crawling the API origin, because /robots.txt there was itself a 404 — which is how /api/v1/... endpoints came to be reported as "Not found (404)". Redirects cannot fix those; they were never pages. The API origin now serves a real robots.txt and sets X-Robots-Tag: noindex on responses for URLs discovered anywhere other than a crawl.

Path mode keeps an Allow: /help/ carve-out. That is the self-host default and it serves help-center pages from the same origin, so a blanket disallow would have de-indexed every self-hosted help center.

The help center itself disallows the paths that are not pages: /feedback and /search, and the ?q= and ?topic= variants of the landing page, which are an unbounded set of near-duplicates. Articles stay crawlable, nothing gets noindex, the sitemap is untouched.

Error responses no longer carry our own exception text

detail=str(e) appears 462 times across the API, so any 500 handed the caller whatever the exception happened to hold — SQL fragments, file paths, connection strings. Patching the seven instances a scan had flagged would have left the pattern everywhere else.

A handler now replaces the detail on any 5xx with a constant and logs the original alongside the route that produced it. 4xx detail is written for the caller and passes through untouched. Four places answer 5xx with something an operator has to act on, such as an upstream "Recipient not on WhatsApp" — those opt out explicitly and are left alone. Future detail=str(e) is safe without anyone having to remember this.

Seven more leaks were ordinary 200 responses rather than exceptions, and so were not covered by that handler: the knowledge listing put the database error in a per-source error field, two status routes returned {"error": str(e)}, and three Shopify GDPR webhooks echoed it back in message. All of them already logged the real reason, so the text was leaking for nothing.

Two unrelated findings from the same scan were worth fixing on their own merits. An agent photo's stored name took its extension from the caller's filename, and now takes it from the content type validated a few lines above, so nothing caller-supplied reaches the path. The Redis TLS check matched .cache.amazonaws.com anywhere in the URL, which redis://evil.com/?x=.cache.amazonaws.com satisfies; it parses the host now.

Self-hosted deployments behind HTTPS

Two bugs that only ever showed up outside our own hosting, both reported and fixed by @spariev.

The dashboard hung at "Connecting to your agent…" on every HTTPS deployment and the backend never saw a handshake. The shipped nginx policy allowed ws://localhost:* but not wss:, and https: does not cover wss: — the browser blocked the Socket.IO connection before it was made. The same policy blocked the Google Fonts stylesheet and its font files. Both are allowed now, along with the origins the Shopify and Meta integrations load from, and web workers.

Separately, app-bridge.js was loaded unconditionally as the first script on every page, and outside the Shopify admin it threw App Bridge Next: missing required configuration fields: shop on every load. It and the Polaris bundle now load only when Shopify's host or shop parameters are present — which also takes about 200 KB of render-blocking script off every dashboard page view that is not inside Shopify.

Notes for self-hosters

• One new migration, idx_kq_org_status_001, indexing knowledge_queue by organization and status. The source limit now reads that table on every file upload and URL add, and it had no index beyond the primary key and priority. Migrations run on backend start.
• The CSP in frontend/nginx.conf has been widened: wss: and ws: in connect-src, Google Fonts in style-src and font-src, cdn.shopify.com across script, style and font, connect.facebook.net for Meta signup, and worker-src 'self' blob:. Every existing allowance was kept, so nothing that worked before stops working. If you front the image with your own reverse proxy, its policy is still the one that applies.
• 5xx response bodies now carry a fixed message. The original error is in the backend log, with the route that produced it.
• The promo offer surfaces added to the dashboard are mount points only. Without the enterprise modules they render nothing and throw nothing.

Testing

29 tests were added for the extraction work, fixture-based and with no network calls, plus 117 lines of new coverage on the knowledge source limit, seven on the 5xx handler, four on the stored upload name — including a filename claiming .php and one with traversal segments — and tests on both halves of the robots work. Thirteen test files changed in total.

One caveat, unchanged by this release: six SSH key tests in test_db_connector_service.py fail on main as well.