[Mixed-source orchestration] Phase 6: Harden and roll out native mixed-source processing

[paullizer]

Parent: #1055

Phase 6 - Hardening, Extraction, and Rollout

Planning baseline version: 0.250.061

Status: Proposed

Depends on: Phases 1-5

Objective

Consolidate mixed-source orchestration into maintainable services, enforce coverage and security invariants, and roll out Chat/Search/Analyze/Compare behavior with production observability.

Scope

1. Coverage ledger

Every requested or planner-selected source must end with one status:

• completed
• partial
• failed
• skipped with an explicit reason

Final synthesis must receive a bounded coverage summary and may not claim complete analysis when the ledger is incomplete.

Required invariants:

• explicit selected sources cannot disappear
• each selected table receives a successful tabular call or terminal failure
• each selected narrative file receives retrieval/window coverage or terminal failure
• source identity includes scope and version
• duplicate filenames never merge identities

2. Failure policy

Define consistent behavior by mode:

Mode	Partial failure behavior
Chat	Answer from available evidence and state the unavailable source
Search	Return available results with per-source omissions
Analyze	Complete as partial only when at least one source succeeds; expose exhaustive coverage
Compare	Skip failed Target pairs; fail if the Source cannot be prepared

Never silently downgrade a table to generic text analysis after tabular failure.

3. Progress and cancellation

Unify mixed-source progress across:

• manifest resolution
• narrative retrieval/windows
• tabular planning/calls/exports
• pairwise comparison
• final reduction

Cancellation must propagate to all active branches and prevent final artifact publication after cancellation.

4. Citations and artifacts

Normalize and deduplicate:

• hybrid search citations
• document-window citations
• tabular tool citations
• generated analysis artifacts
• generated tabular exports

Keep downloadable exhaustive outputs out of model context.

5. Token and cost accounting

Aggregate request counts and tokens across native engines and reductions. Add per-mode latency and model-call metrics without logging source content.

6. Module extraction

Move reusable tabular orchestration out of route_backend_chats.py.

Recommended ownership:

• functions_mixed_source_orchestration.py: manifests, partitions, execution coordination, coverage
• functions_tabular_analysis.py: real reusable tabular runner rather than a lazy route facade
• functions_document_analysis.py: narrative window engine
• functions_document_comparison.py: engine-neutral comparison over evidence envelopes
• route modules: request validation, authorization context setup, response persistence only

Avoid a broad rewrite in one pull request. Move one stable boundary at a time with import-compatibility shims.

Rollout

Feature flags:

• enable_mixed_source_manifest
• enable_mixed_source_chat_search
• enable_mixed_source_analyze
• enable_cross_format_compare
• enable_mixed_source_conversation_continuity

Suggested stages:

1. Development-only telemetry
2. Internal users with explicit selections
3. Analyze mixed selections
4. Cross-format Compare
5. History continuity
6. Default-on after error/latency review

Acceptance Criteria

• Every explicit or planned source has one terminal coverage state.
• No selected table is silently processed as generic narrative text after a tabular failure.
• Mixed-source cancellation prevents later reduction and artifact publication.
• Citations, generated artifacts, token usage, and progress survive mixed branch execution without duplicates.
• Chat/Search remain relevance-bounded while Analyze All Documents is exhaustive within configured limits.
• Personal, group, public, and chat-upload authorization is revalidated at every sensitive boundary.
• Feature flags can disable each mode independently without breaking Phase 1 manifest contracts.
• Production telemetry shows no silent source omissions during the staged rollout.

Performance Gates

• bounded manifest/catalog enumeration
• bounded model and tool concurrency
• no full tabular rows in synthesis context
• no all-document enumeration for Chat/Search
• configurable Analyze document limits
• scheduler fairness for background exports
• no request-lifetime dependency for long-running generated outputs

Security Gates

• object-level authorization for every manifest entry
• reauthorization at background/resume/finalization boundaries
• no caller-supplied scope identity trusted without rebinding
• no raw settings sent to frontend
• no sensitive source descriptors persisted in assistant metadata
• route policy, broken-access-control, and XSS guardrails pass

Tests

End-to-End Test Matrix

Dimensions:

• Mode: Chat, Search, Analyze, Compare
• Sources: narrative only, tabular only, mixed
• Scope: personal, group, public, chat upload, mixed scope
• Runner: model, agent
• Transport: standard, streaming
• Selection: explicit, all, history, relevance
• Outcome: success, partial failure, authorization loss, source version change, cancellation

Minimum release suite:

• PDF + XLSX in all four modes
• CSV + XLSX Compare
• explicit selection with panel closed
• Analyze All Documents beyond top-N search results
• implicit prior-table follow-up
• mixed-scope duplicate filename case
• partial branch failure
• cancellation during each execution phase
• artifact/citation/token aggregation

Observability

Add structured metrics for:

• source counts by kind and status
• engine calls per mode
• missing coverage violations
• partial failure rate
• history reuse/rerun decisions
• mixed synthesis latency
• background export linkage
• cancellation phase
• source authorization failures

Add a diagnostic correlation ID shared by the request, manifest, engine branches, reduction, and artifacts.

Documentation

Update:

• feature documentation for document actions
• Chat/Search behavior documentation
• Compare supported combinations
• admin capability settings
• troubleshooting for partial source coverage
• release notes per phase

Do not expose internal routing instructions in the user interface.

Rollback

Each feature flag rolls back independently. Preserve Phase 1 contracts and telemetry even when a mode-specific behavior flag is disabled.

Exit Gate

The initiative is complete when the full test matrix and security/performance gates pass, production telemetry shows no silent coverage loss, and mixed-source behavior is default-on for Chat, Search, Analyze, and Compare.

Known Future Work

• many-to-many comparison
• richer relational joins across arbitrary workbook sets
• persistent reusable computed-result caches
• user-configurable per-document plus collective Analyze output
• very large catalog planning beyond configured Analyze limits

Not in This Phase

• New document file formats unrelated to mixed-source routing
• A second tabular analysis or export subsystem
• Many-to-many Compare
• Unbounded Chat/Search catalog processing
• Persisting raw table rows in conversation metadata
• Broad UI redesign beyond progress, coverage, and failure states required by this initiative

[paullizer]

Phase 6 implementation is ready for review in PR #1074: #1074

Delivered:

• Manifest-aligned terminal coverage ledger and mode-specific partial-failure policy across Chat, workflow Search, Analyze, Compare, and continuity.
• Cancellation/finalization rollback, fresh authorization and strict version checks, Foundry context opt-out filtering, citation/artifact deduplication, and observed native token aggregation.
• Bounded, approved-access Analyze All backend enumeration behind the existing default-off flag; no mixed-source rollout flags were enabled by default.

Validation: 53 Phase 1-6 mixed-source tests, 67 adjacent integration tests, route policy, BAC/XSS, Python compilation, Pylance diagnostics, JS syntax, and whitespace checks passed.

Remaining rollout gate: live production omission/error/latency evidence and explicit approval before enabling any mixed-source behavior by default.