Make Analyze Markdown-first and generated outputs contract-faithful

[paullizer]

Make Analyze Markdown-First And Generated Outputs Contract-Faithful

Planning baseline version: 0.250.168

Summary

Analyze and Search can now route exhaustive tabular work through the same durable runner, but live validation shows that execution parity does not yet guarantee correct deliverables.

Analyze should always produce a primary Markdown analysis artifact after successful completion. Any explicitly requested CSV, JSON, XML, workbook, DOCX, PDF, or other supported output should be an additional sibling artifact. Structured outputs shared by Search and Analyze must have the same exact public schema, row coverage, ordering, and values.

The first seven implementation PRs established the contract, Markdown-first
admission, public-schema projection, deterministic evaluator, artifact-set
manifest, plural UI, and rollback control. A post-merge audit found that the
roadmap exit gate is not complete, so this issue now tracks the remaining
implementation and final integration work.

Post-Merge Audit Findings

1. Production Search and Analyze adapters do not yet build a transformation
   specification from user rules. The 200-row request therefore still enters
   semantic generation with schema-only validation unless server-side hints are
   injected.
2. Semantic field verification and targeted repair are not implemented.
3. Combined runs stage a structured sibling before Markdown reduction, while
   direct artifact download and promotion authorization do not enforce the
   artifact-set publication generation.
4. A failed run after sibling staging can be stranded by the publication guard
   without compensating cleanup or a safe Continue/cancel path.
5. Multiple requested durable structured formats are still declined.
6. Phase 3 regressed DOCX/PDF requests that explicitly serialize authorized
   current-turn function results.
7. Cumulative scale and lifecycle harnesses need updated isolated helper
   dependencies before the Phase 7 matrix can run.
8. The integration branch must merge the latest Development and pass the
   complete functional, security, UI, restart, lifecycle, and scale gates.

Approved Closure Delivery

• Phase 7A: Stabilization and compatibility

  • Merge current Development into the integration branch.
  • Restore DOCX/PDF function-result serialization without weakening derived
    output or tabular passthrough protections.
  • Repair cumulative scale and lifecycle test harnesses and restore a green
    baseline.

• Phase 7B: Production correctness planning and repair

  • Extend the existing immutable generation plan with a reviewed allowlisted
    transformation graph and deterministic/semantic field ownership.
  • Execute deterministic rules server-side through the shared evaluator.
  • Add independent bounded semantic verification and field-level targeted
    repair with fail-closed exhaustion behavior.
  • Prove the 200-row fixture through the real Search and Analyze adapters with
    no injected output hints and zero value mismatches.

• Phase 7C: Recoverable artifact-set publication

  • Make the validated run-manifest generation the visibility commit marker for
    staged generated artifacts.
  • Enforce object-level lifecycle authorization for download, view, and
    promotion.
  • Add idempotent rollback, cleanup, restart, Continue, and cancellation
    recovery.
  • Fan out one validated canonical row set to all requested durable CSV, JSON,
    and XML siblings in request order.

• Phase 7D: Integration, scale, and rollout evidence

  • Run deterministic/local tiers through 100,000 rows.
  • Run paid live semantic validation through 3,000 rows.
  • Exercise every planning, validation, publication, authorization, restart,
    cancellation, ETag, blob, and browser failure boundary.
  • Complete authenticated desktop/mobile UI and accessibility validation.
  • Retain legacy compatibility code and move only evidence-backed assignment
    to observe or disabled mode.
  • Merge current Development again and open the final integration PR only
    after all blocking gates are green.

A live 30,000-row semantic run and destructive legacy code deletion are not
part of this closure without separate approval.

Observed Behavior

Using a 200-row financial review fixture with explicit row rules and exactly nine requested output columns:

Analyze

• published 200 unchanged source rows
• retained the ten original source columns
• did not produce the requested derived columns
• did not publish a Markdown analysis artifact
• treated foreground source rows as completed generated output

Search/shared durable generation

• produced all 200 transformed rows in order
• included the nine requested columns
• also exposed unrequested source_row_number and source_row_identity columns
• produced five rule-invalid values at FRI-062, FRI-073, FRI-115, FRI-141, and FRI-159

User Impact

• Analyze can attach a file that has the right row count but the wrong meaning.
• Users asking for exact columns receive internal server lineage fields.
• Structural validation can mark an output complete even when generated values violate explicit rules.
• Combined durable runs may create multiple artifacts, but the browser completion path surfaces only one.
• Search and Analyze can appear to have parity while delivering different or incorrect files.
• A user cannot reliably distinguish completed analysis from source-row passthrough.

Root Causes

1. Analyze artifact creation is optional and response-driven instead of an action-level invariant.
2. Structured-export and hierarchical-analysis intent are represented by overlapping booleans rather than one normalized deliverable plan.
3. Combined durable work depends on enable_tabular_hierarchical_analysis; disabled state can silently reduce analysis plus export to export-only behavior.
4. Internal source lineage is inserted into the same schema later serialized for users.
5. Legacy post-tool and generic file finalizers can serialize source/function rows without proving they are the requested derived result.
6. Current batch validation verifies row count, row identity, and field sets, but not prompt-rule correctness.
7. Run status returns plural generated artifacts, while completion polling replaces the progress card with only generated_artifact.

Required Product Contract

• Every successfully completed Analyze action publishes exactly one primary Markdown artifact.
• Requested files are sibling artifacts and do not replace Markdown.
• Search and Analyze structured deliverables have equivalent public schemas and values.
• Internal lineage never appears unless explicitly requested as user data.
• Passthrough is allowed only for explicit unchanged-copy requests.
• Derived output requests require a transformation execution path.
• Deterministic rules use an allowlisted expression graph where supported.
• Semantic outputs receive field-level verification and bounded targeted repair.
• Invalid required output is not published after repair exhaustion.
• Required artifact sets publish atomically or roll back safely.
• Existing runs continue under their persisted contracts.

Target Architecture

flowchart TD
    Q[Authorized request] --> P[Versioned deliverable plan]
    P --> E[Foreground or durable execution]
    E --> V[Structural and semantic validation]
    V -->|repairable| R[Targeted repair]
    R --> V
    V -->|valid| A[Artifact-set publication]
    V -->|invalid| F[Fail without partial publication]
    A --> M[Primary Markdown for Analyze]
    A --> S[Requested sibling outputs]
    M --> U[Plural artifact UI]
    S --> U

Loading

Phased Work

• Phase 1: Baseline and executable deliverable contract

  • Preserve the 200-row fixture as an executable regression oracle.
  • Define action, artifact-role, schema, cardinality, transformation, validation, and publication contracts.
  • Add safe telemetry for source passthrough, extra public fields, semantic mismatches, and singular UI projection.

• Phase 2: Unified intent and Analyze Markdown admission

  • Replace overlapping intent booleans with one normalized deliverable plan.
  • Make Markdown required by Analyze action policy, not prompt wording.
  • Select hierarchical_analysis or combined without silent export-only downgrade.
  • Preserve bounded foreground execution while guaranteeing terminal Markdown publication.

• Phase 3: Public schema, internal lineage, and passthrough safety

  • Split checkpoint lineage from public output fields.
  • Preserve exact requested columns and order.
  • Allow passthrough only when the requested contract is an unchanged copy.
  • Fail closed when derived fields are missing.

• Phase 4: Transformation correctness, validation, and repair

  • Add a constrained deterministic expression graph for representable row rules.
  • Add semantic field verification for non-deterministic work.
  • Repair only failed rows/fields through bounded retries.
  • Refuse publication when required correctness cannot be established.

• Phase 5: Durable artifact-set publication and lifecycle

  • Represent primary and sibling artifacts as one versioned set.
  • Validate and reauthorize every required member before publication.
  • Make completion, retry, cancellation, rollback, and restart idempotent.
  • Keep old run formats compatible.

• Phase 6: Plural artifact UI, persistence, and accessibility

  • Render every completed artifact from generated_artifacts[].
  • Present Markdown first for Analyze, then requested siblings in declared order.
  • Preserve refresh, conversation reload, deduplication, view, download, promotion, cancel, and continue behavior.
  • Validate desktop, mobile, keyboard, screen-reader, and failure states.

• Phase 7: Integration, scale, rollout, and legacy retirement

  • Prove Search/Analyze structured-output equivalence across formats, scopes, and scale tiers.
  • Exercise restart, stale source, authorization loss, partial publication, and UI refresh races.
  • Roll out through shadow and canary modes.
  • Retire unsafe source-row fallback only after telemetry proves replacement coverage.

Acceptance Criteria

• Every successful Analyze fixture has one primary Markdown artifact.
• The 200-row regression CSV contains exactly the nine requested columns.
• The 200-row regression CSV has zero value mismatches.
• Search and Analyze structured results are equivalent after format normalization.
• No internal lineage field appears in public output unless requested.
• No source row set is published as a derived result.
• Every required artifact is visible after completion and reload.
• Partial, stale, unauthorized, failed, or canceled output is never presented as complete.
• Existing durable runs remain operational under their recorded contracts.
• Narrative, mixed-source, per-document, and multi-table Analyze retain correct coverage behavior.
• Personal, group, public workspace, and conversation-upload authorization pass.
• The implementation reuses existing runners, source resolution, storage, and artifact cards.

Related Work

• Refs Scale per-row tabular analysis and generated CSV exports to 30,000+ rows #1031 for durable full-source tabular analysis and generated artifacts.
• Refs Unify mixed-source document orchestration across Chat, Search, Analyze, and Compare #1055 for mixed-source manifests and evidence.
• Refs [Mixed-source orchestration] Phase 3: Make Analyze exhaustive and compositional for mixed sources #1058 for mixed-source Analyze.
• Follows merged PR Add durable tabular Analyze and Search parity #1219 for Analyze/Search durable execution parity.

Planning Package

Detailed phase plans and branch workflow are stored in:

priv-simplechat-2/analyze_artifact_output_contract/

[paullizer]

Phase 7A stabilization is complete and merged through PR #1234.

• Integration parent: 6ed528f6
• Application version: 0.250.178
• Restored explicit Word/DOCX serialization of authorized current-turn non-tabular function results while retaining derived-output, tabular-coverage, and sensitive-field guards.
• Repaired lifecycle and cumulative scale harness dependencies.
• Validation: generated-file suite 35/35; lifecycle 3/3; complete scale matrix through 100,000-row contracts; adjacent Analyze contracts 35/35; Search/Analyze adapters 18/18; syntax, diagnostics, Markdown, and diff hygiene passed.

Starting Phase 7B from the refreshed integration branch for production rule planning, independent semantic verification, and targeted repair.

[paullizer]

Phase 7B correctness slice opened

PR: #1235
Commit: 29232015
Version: 0.250.179

Completed

• Added generation plan v2 with reviewed deterministic/semantic field ownership.
• Persisted normalized tabular-transform-v1 contracts from real Search/Analyze production facades with no injected output hints.
• Executed deterministic fields server-side and proved the 200-row financial-review fixture has exact schema/order/value equivalence with zero mismatches.
• Added independent semantic verification, targeted repair, repair exhaustion, shadow mode, safe aggregate telemetry, and restart-safe candidate checkpoints.
• Preserved v1 plan compatibility and added a new-run fallback deliverable contract for legacy direct preflight when active planning is enabled.

Validation

• py_compile for changed app/test Python files: passed.
• functional_tests/test_tabular_semantic_validation_phase7b.py: 8/8 passed.
• functional_tests/test_tabular_phase7b_production_correctness.py: 2/2 passed.
• functional_tests/test_tabular_transformations_phase4.py: 5/5 passed.
• functional_tests/test_tabular_row_orchestration_scale.py: complete suite passed through 30k bounded finalization and 100k planning/hardening contracts.
• Adjacent Analyze/schema/artifact-set/rollout/background/shared-planner/parity/Search adapter/Analyze adapter suites: passed.
• VS Code diagnostics: clean.
• Broken access control full-file guard on touched Python paths: passed.
• XSS full-file guard on touched Python paths: passed.
• Markdown and diff hygiene: passed.

Still Remaining

• Phase 7C: artifact-set visibility commit, cleanup/recovery, direct route lifecycle authorization, and durable multi-format publication.
• Phase 7D: final integration, chaos, authenticated UI, canary, and approved live semantic scale evidence.

[paullizer]

Phase 7B correctness slice has merged into feature/analyze-artifact-output-contract via PR #1235. Integration parent is now e7fed44a at app version 0.250.179. Next slice is Phase 7C: recoverable artifact-set publication, direct route lifecycle authorization, cleanup/recovery, and durable multi-format publication.

[paullizer]

Phase 7C publication slice opened

PR: #1236
Commit: 4eefd242
Version: 0.250.180

Completed

• Added staged generated-artifact metadata for new tabular artifact-set members.
• Direct generated-artifact download and workspace promotion now reauthorize new members against the completed, validated run manifest and publication generation.
• Legacy generated artifacts without artifact-set metadata remain compatible.
• Artifact-set publication commits backing message metadata only after required members validate.
• Failed post-staging publication states can resume or cancel instead of being stranded by publishing_started_at.
• Durable structured publication can fan out the same canonical checkpoint set to every requested CSV, JSON, or XML sibling in request order.

Validation

• py_compile for changed app/test Python files: passed.
• functional_tests/test_generated_artifact_lifecycle_authorization.py: 6/6 passed.
• functional_tests/test_tabular_phase5_artifact_set_lifecycle.py: 4/4 passed.
• functional_tests/test_analyze_deliverable_contract.py: 6/6 passed.
• functional_tests/test_tabular_background_generated_exports.py: 8/8 passed.
• functional_tests/test_tabular_row_orchestration_scale.py: complete suite passed through 30k bounded finalization and 100k planning/hardening contracts.
• Route policy suites: 12/12 passed.
• Generated artifact promotion modal: 1/1 passed.
• VS Code diagnostics: clean.
• Broken access control and XSS full-file guards: passed.
• Markdown and diff hygiene: passed.

Still Remaining

• Phase 7D: final integration, deterministic scale/chaos matrix, authenticated UI, canary evidence, approved live semantic-through-3k validation, release notes, and final PR to Development.