Consolidate the three progress channels into one

[LegalMarc]

Goal

Collapse the three overlapping progress mechanisms into one channel. The bridge schema migration deliberately deferred this until its pattern had proven out in the simpler shapes; that condition is now met.

Dependencies

None — immediately eligible. The migration steps it waited on (#67, #89, #90, #91, #92, #93, #94) are all closed.

Scope

Three mechanisms carry progress today and they overlap:

1. The rich path, a validated ProgressUpdate pydantic dataclass in src/python/marcut/progress.py, which is the only rich shape since the three-argument fallback was removed in Bridge schema step 4a: validate ProgressUpdate and audit the 3-arg callback path #92.
2. The mass-event path, five pydantic models validated in emit_mass_event and JSON-encoded into the message field, parsed structurally on the Swift side by ingestProgressPayload in DocumentModels.swift.
3. Direct print of the same JSON to stdout, in the same function.

• Design one channel that carries both the phase-level updates and the chunk-level events, and state the design on the issue before implementing it.
• Migrate ProgressTracker and every emit_mass_event call site onto it.
• Update the Swift consumer to match, keeping the progress bar's observable behaviour identical.
• Remove whichever of the three mechanisms the new design makes redundant.

Out of scope

• Changing what progress is reported, how it is weighted, or how the bar animates. This is a transport consolidation, not a behaviour change.
• The intra-chunk streaming design itself, which is settled in its own spike.

Automation

afk — rung 0, but see the risk note. This is the largest remaining bridge item and the one with the most behavioural surface.

Acceptance criteria

• One channel carries progress; the redundant mechanisms are gone from both sides.
• The progress bar behaves identically for a rules-only run, an Ollama run and a llama.cpp run.
• The producer-to-consumer parity pin from Bridge schema step 4b: closed model set for progress events, and the token_progress gap #93 still holds, in both directions, against whatever the new channel's type set is.
• Both suites pass.

Required verification

• source .venv/bin/activate && PYTHONPATH=src/python python3 -m pytest -q tests/test_progress.py tests/test_model_enhanced.py tests/test_pipeline.py
• source .venv/bin/activate && PYTHONPATH=src/python python3 -m pytest -q
• swift test --package-path src/swift/MarcutApp
• swift build --package-path src/swift/MarcutApp
• source .venv/bin/activate && python3 -m ruff check src/python tests
• swiftformat --lint src/swift/MarcutApp/Sources src/swift/MarcutApp/Tests

Notes

• Read docs/design/bridge_schema_migration.md §"Migration Plan" step 4 first. It names this explicitly as the follow-up it will not attempt, and says why: it touches ProgressTracker.__init__'s signature-detection logic and is a larger behavioural change than the migration itself.
• This is the hottest path in the codebase, firing many times per document. Keep the per-event cost at or below what it is today and say so with a measurement.
• Bridge schema step 4b: closed model set for progress events, and the token_progress gap #93's history is the cautionary tale for this ticket. A parity test that derives one side from source but restates the other is not a pin; it only catches drift in one direction. Whatever replaces SWIFT_HANDLED_MASS_EVENT_TYPES, derive both sides.
• token_progress is currently emitted by Python and explicitly ignored by Swift via a documented no-op case. Carry that decision forward deliberately rather than dropping the event or silently starting to consume it.
• If the design turns out to need a call that is genuinely the maintainer's, post the options on the issue and stop rather than guessing.
• Maintainer decision (posted after the coder's design writeup comment above; do not re-ask, implement on this basis):
  1. Wire format: Option B. One discriminated pydantic union owns the schema everywhere (replacing ProgressUpdate and the 5 mass-event models). The in-process PythonKit path keeps reading live attributes off the PythonObject (no serialization cost) branching on the union's discriminator; the stdout/subprocess path keeps using the JSON string. Do not switch the in-process path to always-JSON (that was Option A, rejected — it would add per-event encode/decode cost to the hot in-process path with no measured need).
  2. Fold in both pre-existing bugs found during design. Unify the CLI-vs-PythonKit-bridge arity-detection divergence (the second, undocumented inspect.signature check in model_enhanced.py around lines 1262-1277 that decides whether to build the inner tracker) so both execution paths emit through the same single call shape. Also bring LlamaCppRedactionPipeline.process_document's raw 3-tuple (model_enhanced.py:1957-1959, currently bypassing emit_mass_event and any pydantic model) into the unified schema as a chunk_start/chunk_end pair with no mass_total/keepalive/token_progress — preserving its current, sparser behavior (no new event types for llama.cpp), just routing it through the same validated shape as the Ollama/CLI paths.
  3. Delete dead code. PythonBridge.swift's runRedactionWithCLI/parseMARCUTProgress (PythonBridge.swift:3841-4176) has zero callers in the app target — delete both as part of this ticket rather than leaving them or filing separately.

[LegalMarc]

Design writeup (per ticket instructions: posting before implementing)

I read progress.py, model_enhanced.py, pipeline.py, cli.py,
PythonKitBridge.swift, DocumentModels.swift, DocumentRedactionViewModel.swift
and PythonBridge.swift end to end before drafting this. The system has more
producers and transports than the three named in the ticket body, and the
extra ones change which consolidation design is actually safe. Posting the
full picture and two concrete options rather than guessing, per the ticket's
own escape hatch ("if the design turns out to need a call that is genuinely
the maintainer's, post the options and stop").

Corrected current-state map

Producers (six distinct call shapes feed progress_callback, not three):

1. pipeline._collect_enhanced_spans (pipeline.py:2109-2116) builds its own
   ProgressTracker unconditionally when progress_callback is set, and
   sends two rich ProgressUpdate phase transitions (0.0 and 1.0) for the
   whole LLM_EXTRACTION phase.
2. IntelligentRedactionPipeline.process_document (model_enhanced.py:1194-1252,
   the emit_mass_event closure) validates+serializes one of 5 pydantic
   models and either stuffs the JSON into a second, independent inner
   ProgressTracker's message field, or -- see Keep embedded Python helper executable for Developer ID releases #3 -- falls back to a
   3-positional-arg call.
3. That inner tracker's existence is decided by inspect.signature-based
   arity detection at model_enhanced.py:1262-1277, duplicating (and
   contradicting in outcome) the exact audit Bridge schema step 4a: validate ProgressUpdate and audit the 3-arg callback path #92 already did for
   ProgressTracker.__init__ and removed. For the CLI's callback (a plain
   def callback(update): ... closure), inspect.signature succeeds and
   the inner tracker is built, so _collect_enhanced_spans's outer
   tracker and process_document's inner tracker both fire against the
   same callback. For the PythonKit bridge's callback (a PyCFunction-backed
   PythonFunction, uninspectable), inspect.signature raises ValueError,
   accepts_progress_update comes back False, no inner tracker is built,
   and mass events instead go out via the 3-positional-arg fallback
   (progress_callback(0, 0, display), model_enhanced.py:1238-1245). So the
   two production consumers (CLI vs. the packaged app's PythonKit path)
   already receive mass-event data over different shapes today, undocumented
   by the Bridge schema step 4b: closed model set for progress events, and the token_progress gap #93/Bridge schema step 5: validate metadata settings decoded from the environment #94 migration steps (both only audited ProgressTracker.__init__,
   not this second, separate arity check).
4. LlamaCppRedactionPipeline.process_document (model_enhanced.py:1957-1959)
   never calls emit_mass_event at all -- it calls
   progress_callback(i + 1, total_chunks, f"Processing chunk {i+1}/{total_chunks}")
   directly, an unvalidated 3-tuple with a human string, not JSON. The
   llama.cpp backend therefore has no mass_total/chunk_start/keepalive/
   token_progress events today, only a bare chunk/total pair -- a fourth,
   simpler shape with no pydantic model at all.
5. model.py's on_token_progress (model.py:766-862) feeds
   token_progress into emit_mass_event from inside a try/except Exception: pass streaming loop (Ollama-only; llama.cpp has no
   equivalent).
6. cli.py's own cli_progress_callback (cli.py:150-159) is a sixth
   independent format: MARCUT_PROGRESS: <text> / MARCUT_STATUS: <text>
   lines, printed only for a human reading a terminal. It is not part of
   the ticket's three named mechanisms and not read by the packaged app,
   but it does render update.message (which, per Fix App Store sandbox entitlement for embedded Python helper #2/Keep embedded Python helper executable for Developer ID releases #3, sometimes
   contains a raw mass-event JSON string) as MARCUT_STATUS: {"type": ...}.

Consumers: two real transports exist, not three.

• In-process PythonKit attribute access (PythonKitBridge.swift:1546-1573):
  reads update.phase/.phase_name/.phase_progress/.overall_progress/
  .message off the live PythonObject when args.count == 1, or
  positionally unpacks args[0]/args[1]/args[2] otherwise. This is the
  packaged app's only real path today (DocumentRedactionViewModel.swift
  calls applyPythonKitProgress, which calls item.ingestProgressPayload(message)
  only on whatever ends up in update.message).
• Stdout JSON lines starting with { (ingestProgressPayload,
  DocumentModels.swift:446-527), read by two live call sites in
  PythonBridge.swift (2733, 2819) that spawn run_unified_redaction as a
  subprocess with no progress_callback registered at all -- so in that
  code path, _collect_enhanced_spans's tracker is never constructed,
  phase-level ProgressUpdates never exist, and emit_mass_event's direct
  print(message) (model_enhanced.py:1246-1250) is the only signal that
  reaches Swift. This is why mechanism 3 ("direct print to stdout") can't
  just be deleted -- it's load-bearing for the subprocess path, which is a
  live code path (runRedaction, PythonBridge.swift:2302), not a fallback
  stub.
• PythonBridge.swift's runRedactionWithCLI/parseMARCUTProgress
  (PythonBridge.swift:3841-4176), which is the thing that would consume
  cli.py's MARCUT_PROGRESS:/MARCUT_STATUS: text lines, has zero
  callers anywhere in the app target -- confirmed via
  grep -rn "runRedactionWithCLI". It's dead code, safe to ignore or to
  delete as part of this ticket's cleanup, but not a live consumer.

Why this changes the design space

The ticket's framing ("one channel that carries both phase-level updates and
chunk-level events") reads naturally as "pick one wire shape, use it
everywhere." Two honest options, with different blast radius:

Option A -- one wire shape, always serialized. Fold ProgressUpdate and
the 5 mass-event models into a single pydantic discriminated union
(type: "phase_update" | "mass_total" | "chunk_start" | "chunk_end" | "keepalive" | "token_progress"). One Python-side emission helper validates
and json.dumpss every event, for both trackers and both mass-event paths,
and always both (a) calls progress_callback(json_str) and (b) prints it to
stdout -- collapsing today's attribute-object path, the two 3-arg fallback
paths (#93's message-stuffing and llama.cpp's raw 3-tuple), and the
duplicate-tracker arity check into one call shape with one arity everywhere.
Swift gets one consumer: ingestProgressPayload-style JSONDecoder parsing,
called from both the PythonKit callback (now single-arity, no more
args.count branch in PythonKitBridge.swift) and the subprocess stdout
reader. This is the most literal reading of "one channel" and removes the
most redundancy, but it changes the PythonKit in-process path from zero-cost
attribute access to per-event JSON encode+decode -- previously avoided
deliberately (see docs/design/bridge_schema_migration.md's argument against
paying serialization cost for a boundary that doesn't serialize). Needs the
measurement the ticket asks for to confirm this doesn't regress the hot path
(phase updates are infrequent; the frequent one, token_progress, is
already JSON both ways today, so the actual new cost is limited to
phase-update and chunk_start/chunk_end frequency, which is low -- but the
ticket asks me to say so with a number, not assume it).

Option B -- one schema, two wire shapes. Same single discriminated
pydantic union and single emission helper (still removes the arity-detection
duplication and the llama.cpp bypass), but the PythonKit attribute-access
path stays for the rich in-process case: Swift reads the discriminator off
the live PythonObject (event.type) and branches to attribute access for
phase_update, still avoiding serialization for the hot in-process path,
while the stdout/subprocess path keeps using the JSON string. This keeps
the bridge-migration doc's stated principle (don't serialize an in-process
boundary that already exposes typed live objects) but is a weaker reading
of "one channel" -- it's one schema owned in one place, not one wire
representation, and Swift keeps two branches in ingestProgressPayload's
caller rather than one.

Both options: fix the CLI/PythonKit inner-tracker arity-detection divergence
(item 3 above) by removing the second inspect.signature check entirely --
process_document stops deciding on its own whether to build a tracker and
instead always emits through the single new call shape. Both also bring
llama.cpp's raw 3-tuple (item 4) into the same validated schema as a
chunk_start/chunk_end pair with no mass_total/keepalive/
token_progress (preserving its current, sparser behavior -- the ticket's
"not a behavior change" constraint). cli.py's human-readable
MARCUT_PROGRESS:/MARCUT_STATUS: printer (item 6) is untouched either way
since it's a human terminal format, not part of the Swift-facing bridge --
unless you'd rather I fold it into the same helper for consistency, which
would touch a fourth consumer (a human running marcut on the command line)
that acceptance criteria doesn't mention.

What I need from you

1. Option A or Option B for the wire representation. I lean B (respects
   the standing architectural principle from the bridge-migration doc, no
   forced serialization on the in-process path) but A is the more literal
   "one channel" and is simpler to keep in parity going forward (one Swift
   consumer instead of two).
2. Confirmation that fixing the arity-detection divergence (item 3) and
   folding llama.cpp's bypass (item 4) into the unified schema are in
   scope for "removing the redundant mechanisms" here, rather than out of
   scope under "not a behaviour change" -- I read them as bugs the
   consolidation naturally fixes as a side effect (each already produces
   observably different Swift-side data for the same logical event
   depending on which of the app's two execution paths is running), not as
   a deliberate behavior this ticket should preserve, but they are new
   findings outside the ticket's original three-mechanism description and
   I'd rather confirm than fold them in silently.
3. Whether runRedactionWithCLI/parseMARCUTProgress
   (PythonBridge.swift:3841-4176, confirmed zero callers) should be
   deleted as part of this ticket's cleanup or left alone / filed
   separately.

Holding off on implementation until I hear back, per the ticket's own
instruction to stop rather than guess on a call this size.

[LegalMarc]

Autonomous loop parked this ticket.
Coder blocked: The wire-format choice (JSON-always vs. schema-shared/attribute-access-kept) and whether to fold in two newly-found pre-existing inconsistencies are genuinely the maintainer's call, and the ticket explicitly says to post options and stop rather than guess.

[LegalMarc]

Maintainer decision

1. Wire format: Option B — one discriminated pydantic union owns the schema everywhere; the in-process PythonKit path keeps reading live attributes off the PythonObject (no serialization cost), the stdout/subprocess path keeps using the JSON string. Matches the bridge-migration doc's principle of not paying serialization cost on a boundary that already exposes typed objects.

2. Fold in both pre-existing bugs. Unify the CLI-vs-PythonKit arity-detection divergence (item 3) and llama.cpp's raw 3-tuple bypass (item 4) into the new single schema as part of this ticket — they're bugs the consolidation naturally fixes, not behavior to preserve.

3. Delete the dead code. runRedactionWithCLI/parseMARCUTProgress (PythonBridge.swift:3841-4176, zero callers) gets deleted in this ticket rather than left alone or filed separately.

Proceed with implementation on this basis.