Add general implicit output inversion

[yilibinbin]

Summary

• replace narrow implicit-fit shortcuts with a general target-aware output inversion initializer
• preserve output-space residuals/statistics and branch-aware implicit solving
• fix self-consistent fit worker payloads for uncertainty-bearing sigma rows
• merge current main and remove stale backend toggles/legacy implicit strategy tests

Verification

• QT_QPA_PLATFORM=offscreen PYTHONPATH=. pytest -q -x $(git ls-files 'tests/*.py') -> 1253 passed, 9 skipped, 1 warning
• ruff check touched/relevant files -> passed
• python -m compileall -q data_extrapolation_gui.py app_desktop shared tools fitting cli tests -> passed
• git diff --check -> passed

Notes

• broad ruff across the entire tree is still blocked by pre-existing unrelated lint debt in shared/pdf preview, shared/init.py star re-export, and older tests.

Summary by CodeRabbit

Release Notes

• New Features

  • Added MCMC refinement for implicit-model fits, providing credible intervals and acceptance diagnostics
  • Introduced output-inversion initialization to improve implicit-model convergence

• Enhancements

  • Workspaces now preserve numeric-mode settings for constant editors across save/load cycles
  • Enhanced implicit-fitting diagnostics and error handling

• Refactoring

  • Simplified implicit-fitting backend; removed legacy affine-output and seed-hint strategies
  • Restructured implicit-model solving for improved robustness and maintainability

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

This PR systematically replaces narrow implicit-fit optimizations with a unified, target-aware output inversion initializer; refactors implicit model solving with branch-aware caching; restructures fit job execution to support MCMC refinement and subprocess handling; and updates the runner to route all non-observed-linear implicit fits through the new inversion layer with comprehensive test coverage.

Changes

Implicit Fitting Architecture Overhaul

Layer / File(s)	Summary
Output Inversion Detection and Candidate Generation fitting/output_inversion.py, tests/test_output_inversion.py	New OutputInversion API parses implicit model output expressions, performs symbolic solving with timeout and candidate limits, validates candidates via derivative checks and forward reconstruction, falls back to numeric root-finding with dataset-level budget enforcement, and includes hardened SymPy deserialization with AST-based safety validation.
Implicit Strategy Simplification and Classifier Updates fitting/implicit_classifier.py, fitting/implicit_planner.py	Removes OBSERVED_NONLINEAR from ImplicitStrategy enum and routes non-linear observed equations to GENERAL. Deletes the plan-based routing system (ImplicitPlanKind, ImplicitPlan, plan_implicit_fit) that previously selected between multiple optimization paths.
Implicit Model Caching and Branch-Aware Solving fitting/implicit_model.py, tests/test_implicit_model.py	Refactors ImplicitEvaluationCache to include target_implicit_candidates and branch-aware cache keying; reworks _solve_implicit_value to validate solutions against target output branches and deduplicate seeds; removes analytic derivative paths; and simplifies observed-implicit linear statistics computation with direct metric calculation.
Runner Integration of Output Inversion and Implicit Strategy Selection fitting/runner.py, tests/test_implicit_performance_regression.py, tests/test_example_workspaces.py	Restructures FitRunner._fit_self_consistent to classify via ImplicitProblemClassifier, conditionally fit observed-linear fast path, detect output inversion, seed parameters via inversion-derived candidates using fit_custom_model with a model_factory, and populate implicit_diagnostics/implicit_strategy details. Removes SciPy benchmarking and analytic derivative routing.
Removal of Legacy Implicit Strategy Modules fitting/implicit_derivatives.py, fitting/implicit_seed_hints.py, fitting/implicit_transforms.py	Deletes modules providing analytic derivative evaluation, inverse-square seed hint detection, and affine output transform detection, consolidating all initialization logic into the unified output inversion layer.

Fit Job Serialization and Subprocess Execution for MCMC

Layer / File(s)	Summary
Fit Job Serialization for Sigma and Implicit Model Definitions app_desktop/workers_core.py, tests/test_app_desktop_workers_core.py	Updates FitJob.sigma_rows to support UncertainValue elements; rewrites implicit model definition serialization with explicit payload type validation; adds structured sigma serialization for uncertain values; extends parallel config to accept process_start_method with validation; and adjusts numeric stringification fallback.
MCMC Refinement with Multiprocessing Support app_desktop/workers_core.py, tests/test_mcmc_pre_flight_health.py	Adds _attach_mcmc_refinement that constructs log-probability from evaluator predictions over observations/targets, samples only free parameters, validates evaluator arity via inspect.signature, records medians/credible intervals/acceptance fraction and optional corner-plot PNG to best_fit.details, and surfaces multiple failure modes via mcmc_warning.
Subprocess Execution with Legacy Hook Detection app_desktop/workers_core.py, tests/test_app_desktop_workers_core.py	Introduces queue-based subprocess entry point, robust termination helper with multi-stage cleanup, legacy subprocess runner with polling and deadline handling, and hook-replacement detection to conditionally route to legacy implicit fitting strategies vs. new FitRunner path.

Type Annotations, Model Factory, Workspace, and Test Updates

Layer / File(s)	Summary
Type Annotations and Model Factory Support fitting/__init__.py, fitting/hp_fitter.py	Tightens build_parameter_state type annotation for parameter_names to Any | None; adds optional model_factory parameter to fit_custom_model that rebuilds model specifications during fit execution; updates hp_fitter to compute gradients and statistics using dynamically rebuilt current_model.
Workspace Numeric Mode Persistence and Infrastructure app_desktop/workspace_controller.py, app_desktop/update_controller.py, fitting/model_selector.py, tests/test_workspace_controller.py, tests/test_auto_fit_cancellation_and_timeout.py	Adds numeric_mode capture/restore for constants editors and implicit constants in workspace round-trips; updates model selector timeout handling with precision-state guards via a shared timed_out flag; refactors Qt thread cleanup to deferred reference clearing via QTimer.singleShot.
Comprehensive Test Suite Updates for New Architecture tests/test_app_desktop_workers_core.py, tests/test_implicit_model.py, tests/test_implicit_d8_runner_regression.py, tests/test_fitting_problem_boundary.py, tests/test_symbolic_math.py, plus 12+ test files	Rewrites implicit-fitting, worker, workspace, and example test files to validate output inversion detection, branch-aware caching, inversion-seeded parameter configs, subprocess execution routing, MCMC refinement inputs, numeric mode persistence, and removal of legacy implicit strategy modules.
Implementation Plan Documentation docs/superpowers/plans/2026-05-31-datalab-general-implicit-inversion-plan.md	Comprehensive architectural specification detailing output inversion design, removal of narrow optimizations, branch-aware caching semantics, dataset-level inversion failure behavior, test strategy, and verification steps.

Possibly Related PRs

• yilibinbin/DataLab#45: Modifies subprocess execution and timeout_seconds handling in app_desktop/workers_core.py, directly related to this PR's subprocess execution refactor.
• yilibinbin/DataLab#47: Overhauls implicit/self-consistent backend routing in fitting/runner.py, directly aligned with this PR's runner refactor for output inversion integration.
• yilibinbin/DataLab#46: Adds process_start_method to parallel_config and subprocess termination helpers, overlapping with this PR's fit job serialization and subprocess execution changes.

🐰 From implicit hints to inverted truth,
We solve for candidates with numerical proof,
Branch-aware cache holds secrets tight,
While subprocess workers refine through the night! ✨

🎯 4 (Complex) | ⏱️ ~60 minutes

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/general-implicit-inversion