fix(llm-worker): isolate cycle stages so one failure stops discarding the others' work - #1749
Merged
Conversation
… the others' work run_once() called its three stages unguarded. When analyze_daily_report() raised -- which it did every cycle from 2026-08-22, on a schema Ollama's grammar compiler rejected -- the whole cycle aborted. The sessions and payloads already analysed that cycle were thrown away, and the status document reported nothing but the exception type. That is what made a broken daily report look like a broken worker, and why LLM_DAILY_REPORT_ENABLED=false was the only workaround available: turning the report off was the only way to get the other two pipelines running again. The three stages share one flaky dependency -- a model endpoint -- so any of them can fail transiently. Each now runs isolated: a failure is recorded in stage_errors and the cycle is reported not-ok, but the other stages still run and their results still count. Only the exception type is recorded, never its message. These exceptions come from a model fed attacker-controlled text, and a message can carry that text back out into the status file; the outer cycle handler already applied that reasoning and this keeps it. On the grammar failure itself: it no longer reproduces. The real DailyReport.model_json_schema(), byte-for-byte what worker.py sends, is accepted by the deployed Ollama 0.32.13 at every realistic prompt size (500 to 20,000 characters), as are SessionAnalysis and PayloadAnalysis and every trimmed variant of DailyReport. Notably the Ollama container has not been restarted since before the failure was first seen, so this is not a version change -- which means the trigger was environmental and could return. That is the case this change exists to survive: re-enabling the report can no longer cost the session and payload pipelines anything. Refs #1748
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
Xore
added a commit
that referenced
this pull request
Aug 23, 2026
…ive-only overlay (#1752) Captured-data mode was authorized for this fleet on 2026-08-22 (#1660) by applying docker-compose.captured-data.yml to the running stack by hand. That change lived only in the running container. It was in no file, so the Arcane gitops sync -- which materializes llm-worker/docker-compose.yml -- silently reverted it on the next redeploy. The result is worse than falling back to synthetic mode. The env vars survive, so the worker still believes it is in captured-data mode; it simply cannot reach anything: elastic_transport.ConnectionError: Failed to resolve 'elasticsearch' llm-worker: cycle failed safely: RuntimeError Nothing says "the overlay is gone". I hit this deploying #1749 and only caught it by checking the container after my own redeploy. docker-compose.yml's header is right that the mounts and networks must not be folded into it -- #83 owns that authorization and the base file has to stay safe by default for anyone running it directly. So nothing is folded. Both files are untouched; `include` composes them at deploy time, and the Arcane sync's composePath points at this file instead. That makes the authorization a reviewable artifact rather than undocumented state in a running container. Deleting this file returns the deployment to synthetic-only, which is what makes it a record of the decision rather than a workaround. Verified against the real files on the homeserver: the merged config keeps name: llm-worker, carries LLM_ALLOW_CAPTURED_DATA and the payload mounts, and attaches all three networks including the external honeypot-llm and honeypot-llm-data that the base file alone does not. Closes #1751
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #1748
The structural bug
run_once()called its three stages unguarded. Whenanalyze_daily_report()raised — which it did every cycle from 2026-08-22 — the whole cycle aborted. The sessions and payloads already analysed that cycle were thrown away, and the status document reported nothing but the exception type.That's what made a broken daily report look like a broken worker, and why
LLM_DAILY_REPORT_ENABLED=falsewas the only workaround: turning the report off was the only way to get the other two pipelines running again.The three stages share one flaky dependency — a model endpoint — so any of them can fail transiently. This isolates them: a failure lands in
stage_errorsand the cycle is reported not-ok, but the other stages still run and their results still count.Only the exception type is recorded, never its message. These exceptions come from a model fed attacker-controlled text, and a message can carry that text back out into the status file. The outer cycle handler already applied that reasoning; this keeps it.
On the grammar failure itself — it no longer reproduces
I tried to reproduce it before fixing it, and could not. Tested against the deployed Ollama:
DailyReport.model_json_schema(), byte-for-byte whatworker.pysendsSessionAnalysis,PayloadAnalysismaxLength, one array, three arraysThe earlier bisect's report that "a summary string plus a single enum reproduces it" also does not hold now — that shape passes, including at
maxLength: 2000.The Ollama container has not restarted since before the failure was first seen (0.32.13, up 6 days), so this is not a version change. The trigger was environmental and could return.
That is precisely the case this PR exists to survive. I'm deliberately not claiming the grammar bug is fixed — only that it is not currently reproducible, and that re-enabling the report can no longer cost the session and payload pipelines anything if it comes back.
Verified
29 tests pass, including four new cases: a failing stage keeps the others' results, a healthy cycle reports no
stage_errors, only the exception type is recorded (asserting the attacker-controlled text does not leak into the status), and isolation applies to every stage rather than just the report.Next
Re-enable
LLM_DAILY_REPORT_ENABLEDon the live worker once this is deployed, and confirm a real report is written to thellm-analysisindex — the one annotation pipeline that has never produced a single real output.