[Mixed-source orchestration] Phase 1: Add authorized source manifest and evidence contracts

[paullizer]

Parent: #1055

Phase 1 - Source Manifest and Evidence Contracts

Planning baseline version: 0.250.061

Status: Proposed

Depends on: None

Objective

Create the shared, authorization-safe contracts required to route mixed document selections without changing the final user-facing behavior yet.

This phase removes the architectural assumption that all selected documents must use the same processing engine.

Current Defect Addressed

functions_workflow_runner.py::_resolve_tabular_document_action_documents(...) returns an empty list when any selected document is unresolved or non-tabular. A Word/PDF file therefore suppresses all selected CSV/XLSX files.

Other paths repeatedly resolve the same IDs into different shapes, which allows file classification, scope, and source identity to drift between Search, Analyze, Compare, and tabular processing.

Scope

1. Authorized source manifest

Add a shared resolver that returns one ordered manifest entry per requested source:

{
  "document_id": "...",
  "display_name": "...",
  "file_name": "...",
  "extension": ".xlsx",
  "source_kind": "tabular",
  "scope": "personal|group|public|chat",
  "scope_id": "...",
  "group_id": null,
  "public_workspace_id": null,
  "conversation_id": "...",
  "source_version": "...",
  "authorization_status": "authorized"
}

Required source kinds:

• tabular: CSV, XLS, XLSX, XLSM
• narrative: PDF, DOC, DOCX, PPT, PPTX, TXT, Markdown, and other chunk-backed formats
• unsupported: selected item exists but has no supported native processor
• unresolved: ID could not be resolved or authorized

The resolver must preserve request order and reject duplicate IDs without reordering the first occurrence.

2. Capability partitioning

Add a pure partition helper that returns:

• tabular_sources
• narrative_sources
• unsupported_sources
• unresolved_sources

Non-tabular sources must be filtered into their cohort, not treated as a reason to return an empty tabular set.

3. Shared evidence envelope

Define one engine-neutral result shape:

{
  "document_id": "...",
  "source_kind": "tabular|narrative",
  "engine": "tabular_tools|document_analysis|hybrid_search",
  "status": "completed|partial|failed|skipped",
  "summary": "...",
  "evidence": [],
  "citations": [],
  "generated_artifacts": [],
  "coverage": {},
  "error": null
}

The envelope must remain internal and bounded. Exhaustive row output stays in generated artifacts/checkpoints rather than synthesis context.

4. Selection contract

Define selection_mode values for later phases:

• selected: explicit document IDs
• all: authorized catalog requested by Analyze
• history: no current selection; reuse prior grounded sources when needed
• relevance: Search/Chat catalog candidate selection

This phase defines and validates the shape but does not yet change Chat behavior.

5. Diagnostics

Add structured logs for:

• requested source count
• resolved source count by kind
• unresolved/unauthorized count
• duplicate IDs removed
• selected scope distribution
• manifest resolution duration

Do not log content, credentials, or sensitive source paths.

Proposed Ownership

Create a focused module such as:

application/single_app/functions_mixed_source_orchestration.py

It should depend on existing authorization-aware resolution helpers and expose pure contracts to Chat, workflows, Analyze, and Compare. It must not import route functions.

Acceptance Criteria

• Resolving report.docx + data.xlsx returns one narrative and one tabular manifest entry.
• Reversing file order produces the same partition while preserving requested order inside the manifest.
• One unsupported file does not erase valid sources.
• Unauthorized group/public sources fail closed and appear as unresolved coverage entries without exposing their metadata.
• Duplicate filenames in different scopes remain distinct by source identity.
• Duplicate IDs are processed once.
• Personal, group, public, and chat-upload sources use current authorization checks.

Tests

Add executable tests for:

• mixed extension classification
• order permutations
• duplicate IDs and duplicate filenames
• unresolved source among valid sources
• authorization loss for group/public sources
• chat-upload resolution using conversation ownership
• evidence envelope serialization and size bounds
• selection_mode normalization

Update existing tabular document-action tests so they no longer accept all-or-nothing resolution as valid behavior.

Rollout

Introduce the manifest behind an internal setting such as enable_mixed_source_manifest. In this phase, existing callers may log/compare the manifest while still using legacy execution.

Rollback

Disable the setting. No persisted data migration is required because the manifest is request-scoped.

Exit Gate

Proceed when the manifest has executable authorization/classification tests and can be produced for Chat and workflow requests without changing their current response behavior.

Not in This Phase

• Running either analysis engine
• Changing frontend selection behavior
• Collective synthesis
• All Documents enumeration
• Follow-up source reuse

[paullizer]

Completed by #1064 and merged into ix/1031-tabular-row-orchestration-scale as 28d4700. Phase 1 added the shared authorization-safe ordered source manifest, mixed-source partition contract, validated selection modes, bounded evidence envelope, privacy-safe diagnostics, projected chat-upload metadata resolution, and default-off shadow integrations without changing Chat/Search/Analyze/Compare execution. Application version: 0.250.062. Validation included 14 focused behavioral tests, Python compilation, 12 route-policy checks, full-file broken-access-control and XSS scans, editor diagnostics, and whitespace checks. Parent initiative: #1055.