v0.9.0

Target: optional Spark execution and MLflow integration for Skyulf Core.
Development version: 0.9.0. This release has not been published; Spark node
execution, MLflow registry and Databricks delivery remain under development.

Changed

• Nested tuning now runs independent inner searches within outer training folds,
  with preprocessing refit inside each fold and a separate final model search.
  Optional inner-fold controls and saved outer evaluation results are connected
  across Core, Canvas and Databricks. Search budgets apply per search; nested
  threshold tuning is rejected until it can be evaluated within outer folds.

• Logistic Regression preserves explicit L1/L2/Elastic Net/unpenalized settings
  on older scikit-learn runtimes. The newer l1_ratio/C translation is applied
  only when the installed estimator uses that API, keeping tuning refits and
  ensemble components consistent with the requested penalty.

• Databricks quality-gate model-version tags now use Unity Catalog-compatible
  names. Post-selection ensemble CV removes the completed tuning control while
  preserving the selected component parameters.
  Automatic halving spaces leave the selected resource to the scheduler, and
  saved-model actions no longer expand editable training search recipes.

• Databricks ensemble recipes live in optional src/ensemble.py, separate from
  common search configuration. Named weights, calibration, stacking meta-learners
  and fixed component parameters are validated without silent Core fallbacks.
  Hard voting now supports prediction-only local artifacts and matching MLflow/SDK
  schemas. SGD is available as an ensemble component, matching the Canvas picker.

• Databricks setup now selects a Core tuning strategy directly, with automatic
  model search spaces, task-specific models/ensembles and default/custom controls.
  Shared CV refits preprocessing within training folds, preserves final holdout
  isolation and saves selected parameters/trials. Optional src/tuning.py
  overrides and bounded SHAP evidence are captured with the fitted artifact.
  Standalone tuning now prepares ensemble members and excludes temporal ordering
  metadata from required inference inputs; halving forwards resource bounds.

• Databricks model comparison supports optional absolute quality_gates
  alongside one selection metric. First champion and manual approval enforce
  every gate, with per-gate notebook/MLflow evidence and unchanged historical
  single-gate receipt hashes. Comparison validation and decisions now use
  focused helpers with explicit absolute-improvement semantics.

• Databricks lifecycle tasks restore saved custom preprocessing registrations
  before validating the training specification or running CV in a new process.
  Pinned recipes remain authoritative across task boundaries.

• Generated Databricks scoring uses a fixed run_score_notebook entrypoint
  without a training artifact directory. The role-based notebook API remains
  a compatibility adapter; fixed lifecycle phases and SDK APIs are preserved.

• Generated Databricks projects now carry a shorter setup/operator README and
  minimal preprocessing hooks. Custom Calculator/Applier tutorials live in the
  shared examples directory; saved single-file code remains replayable. README
  graph-contract instructions now match the generated lifecycle tasks.

• Databricks basic setup now uses defaults for advanced snapshot/split, sampling,
  CV and policy details, reducing default prompts from 33 to 26. Existing init
  files retain their values and validation; the setup guide explains overrides.

• Databricks comparison/decision tasks avoid a discarded source replay and reuse
  verified comparison metadata within its phase. Registration still rechecks its
  receipt immediately before mutation; decision retains fresh evidence and
  membership checks before alias changes.

• Databricks evidence and training-spec validation now use focused helpers;
  lifecycle completion and failure handling are separated while preserving
  validation order, durable receipts and original errors.

• Databricks lifecycle replay and approval now share saved training-spec parsing
  and verified evidence loading. Common result construction lives in the workflow
  layer, removing the task adapter's notebook dependency while preserving imports
  and operator/scoring output.

• SDK and durable Databricks training now share policy/CV validation, source and
  champion preparation, and bounded registry tags. Task receipts, SDK callbacks
  and existing champion compatibility checks retain their execution boundaries.

• First-candidate notebook reports now render a missing champion's null metrics
  as "No champion", preserving readable comparison and approval/rejection output.

• The Bundle lifecycle graph now has eight tasks: training/evaluation/registration
  and comparison/promotion each share a task. finalize_and_report joins both
  branches with ALL_DONE, closes failed training, and publishes a score request
  only after verifying successful saved evidence. Redundant edges and standalone
  bookkeeping notebooks were removed; the two serialized jobs are preserved.

• Bundle training uses one train action for manual and cron invocations;
  train_monthly has been removed. An unset training_version resolves the
  latest source snapshot once, while explicit versions remain pinned. Data
  windows remain independent of schedules, and explicit result cutoffs are
  preserved. Regenerate existing scheduled jobs with the matching wheel.

• The local Bundle exposes preparation, training/registration and comparison/decision
  as real tasks in the same serialized lifecycle job.
  Manual actions bypass training; tasks share pinned MLflow artifacts and
  verified phase evidence. Training finalization is independent of scoring,
  and the final decision/operator inputs appear in finalize_and_report.
  Regenerate the graph and notebooks with the matching wheel (graph contract 2).
  Lifecycle repairs/retries are rejected until uncertain effects are inspected.

• Local candidate training records its pinned source/window and original pipeline
  input in MLflow before reading data, splitting, CV or fitting. Runtime failures
  retain replay evidence on a FAILED run without publishing a model. Manual
  replay uses the original config so fixed cleanup is not prepended twice.

• Bundle training and scoring have independent optional cron, timezone and pause
  settings; selected schedules default to enabled. Rolling temporal training can
  choose its holdout duration, and result filtering can use a configurable delay
  in elapsed UTC hours. Manual replay retains explicit data boundaries.

• Databricks source periods compare UTC instants across repeated DST hours.
  Controlled champion reads verify the committed registry event; concrete model
  versions and registry URI types fail early when invalid. Concurrent MLflow
  experiment creation reuses the winning experiment without hiding other errors.
  Pre-split execution and preservation checks are separated into focused helpers.

• Fixed pre-split admission uses per-node validators and derives its supported
  node set from the same rule table. Validation behavior and errors are unchanged.

• Bundle pre-split recipes support validated fixed cleanup, deterministic
  Deduplicate and explicitly declared custom row filters in preprocessing.py.
  Saved models apply feature cleanup once to raw prediction inputs; target-only
  cleanup and training eligibility filters do not require prediction columns.
  Approval restores saved custom code before verifying the recipe and evidence.

• Text vectorizers preserve fitted feature columns and dtypes on empty evaluation
  partitions. TF-IDF and hashing recipes can now train through the local Bundle
  adapter without failing on its empty test partition, on pandas and Polars.

• Local retraining saves versioned cleanup evidence with the recipe, saved Python
  source identity, ordered survivor/train/holdout membership and row counts.
  Automatic and manual lifecycle replay verify this evidence against the model
  artifact; editing the current project file cannot replace its saved recipe.
  Training eligibility filters remain separate from prediction inputs.

• Bundle projects can define build_pre_split_steps() beside Python preprocessing
  to filter training rows with Core DropMissingRows and ManualBounds before
  splitting. Bounded reads and optional source samples stay ahead of cleanup;
  filter-only columns stay outside model inputs. Saved recipes and exclusion
  counts support the same holdout replay during model approval. Scoring does not
  apply training eligibility filters.

• Core lag/rolling now reject a missing declared sort column, and direct lag
  artifacts reject nonpositive/noninteger lags. Known current-target rolling is
  blocked regardless of split placement; backend admission reuses the Core rule.
  Temporal artifacts remain configuration-only, with caller-supplied history.

• Bundle date questions follow user-declared column representation without reading
  source data. Native timestamps skip parsing questions; date-only and text choices
  reveal only applicable format/timezone/midnight rules. Disabled date use, CV,
  scheduling and policy compute hide their dependent questions.

• Generated Bundle notebook tasks disable automatic retries, including the
  separate serverless auto-optimization retries that can register an extra
  candidate after a failed training task. Recovery remains explicit.

• Saved local model evaluation uses sequential joblib prediction to reproduce
  parallel forest metrics during exact registry-evidence replay. Training and
  ordinary batch inference retain their configured parallelism; approval checks
  and receipt digests remain exact.

• Bundle initialization is grouped into data, model, evaluation
  CV, lifecycle and compute sections. Window/split choices, source date parsing
  and training sampling are configurable. A generated offline preview reuses
  job preflight and Core registry metadata; model parameters remain editable
  in workflow JSON and ordered preprocessing in a project Python file.

• Bundle setup accepts comma-separated key/feature names, shows task-specific
  model/metric menus and explains dates, data snapshots, sampling and CV with
  examples. Users can choose the prediction output table name. scheduled
  enables the selected training cron after deployment, including dev; arbitrary
  cadence remains independent of data-window selection. Old init files must use
  the new input field names.

• Generated src/preprocessing.py defines standard Skyulf steps and optional
  custom Calculator/Applier classes. Training saves the self-contained source
  with learned state; local/MLflow inference restores that exact code. Custom
  steps refit within CV folds. Editing the project affects new training, not
  existing-model scoring or lifecycle actions. Required packages must already
  be installed; sibling project modules are not automatically packaged.

• Local Bundle training can run Core CV with fold-local preprocessing on pandas
  or Polars, recording separate MLflow CV evidence without using the final holdout.
  Explicit seeded Spark-side sampling pins membership before bounded transfer.
  Full/fixed/rolling source windows are independent of evaluation splitting and
  cron settings; rolling boundaries use a declared business timezone.

• Bundle input limits use max_input_mb (default 64 MiB) instead of a byte-valued
  field. Training, scoring and approval share explicit conversion to the SDK
  byte budget. Oversized inputs still fail without implicit sampling.

• Local candidate training supports explicit random or temporal holdouts with
  independent optional result-availability filtering. Date-free mode reuses Core
  splitting with stable keys; saved snapshot/settings and holdout membership
  evidence protect replay. Bundle defaults need no invented timestamps, and
  monthly random training reads a bounded pinned snapshot without a lookback.

• Training date rules explicitly declare source formats, timezones and date-only
  interpretation. Normalize before Spark filtering, reject ambiguous/invalid
  dates, retain instants across driver transport, and save the same parsing rules
  for candidate approval replay. Native timestamps keep a simple default path.

• Use record_key_columns and result_available_at_column directly across
  Bundle settings, training specs and inference contracts. This pre-production
  rename removes the old field names without aliases; regenerate projects and
  retrain existing experimental models before using the updated library.

• Bundle configuration has a versioned, task-aware preflight and explicit
  legacy migration. Initialization exposes source/column/composite-key mappings
  and bounded reads, leaving manual snapshots unset. The score job accepts a
  run-only model version override without redeployment; reports distinguish
  the selected version from the model recorded by a previous prediction write.

• Bundle notebooks show shared readable result summaries, metric comparisons
  and next-action tables while preserving their JSON API result. Approval and
  rejection resolve full comparison proofs from committed registry receipts,
  removing manual hash copying without weakening Core evidence checks.

• Generated Bundles expose independent score selection, promotion policy and
  optional score handoff. The existing serialized lifecycle job accepts
  approve/reject/rollback with explicit evidence, returns copyable next-action
  inputs and conditionally calls the score job. Two fixed-role entrypoints
  delegate to a shared Core adapter; no third job or control table is added.

• Score notebook entrypoints ignore lifecycle evidence automatically forwarded
  by Databricks Run Job tasks. Fixed score dispatch and explicit role/action
  override guards remain enforced. An operator walkthrough documents policies,
  approval, rejection, rollback and prediction behavior with Mermaid diagrams.

• Model lifecycle retains displaced contenders as previous_challenger,
  without rotating history on repeated nomination. Promotion preserves it;
  rollback or renomination clears a history pointer that becomes a current
  role. Checked event records and pending-write protection cover history too.

• Local workflow adds explicit reject and rollback actions without retraining.
  Rejection preserves evaluation results and blocks implicit reconsideration;
  unchanged retries reuse checked rejection/rollback receipts. Scoring pins
  remain unchanged; personal serverless Bundle verification passed.

• Core-owned tests now live under skyulf-core/tests, including the Bundle
  template/notebook/lifecycle tests. Backend and frontend contract tests retain
  their own scope; moved tests also participate in Core's type checks.

• Local workflow approval loads an existing candidate's saved comparison and
  training snapshot, rechecks quality and returns a checked promotion receipt
  without training or registering another model. Repeated current approvals
  return the original receipt; live Bundle approval/rejection/rollback passed.

• Core's local workflow accepts independent scoring selection and promotion
  policies. All four combinations preserve scoring pins and promotion gates;
  partial/mixed configurations fail early. Legacy modes retain their behavior.
  Bundle prompts and operator-action handoff use these independent policies.

• Bundle initialization accepts an optional business risk label and documents
  the PayingRegNo cost-tag choice. New lifecycle receipts use from_version;
  older from and compact receipts remain readable.

• Generated Bundle notebooks delegate to reusable local workflow and prediction
  output services in Core. Training, both scoring policies and target checks
  retain their behavior; the notebook only handles widgets, JSON and results.

• Training uses separate company-compatible data/model tags and an optional
  risk category. Dataset identity moves to a run artifact; validation reasons
  and lifecycle receipt fields use readable English, retaining old receipt support.

• Bundle training nominates every registered contender as challenger before
  comparison, retains rejected contenders with validation status/reason, and
  records comparison failures. Promotion remains gated; pinned scoring stays
  unchanged. Rollback preserves a separately verified contender. Generic Core
  training opts into this behavior through an explicit registration callback.

• The local-engine Bundle template exposes editable dev/test/syst/prod
  targets with separate host/catalog bindings and a serverless or policy-backed
  compute choice. Its source lives in skyulf-core/templates/databricks.
  Its default now deploys only train and score; the first
  score validates the source and pinned model and creates only its prediction
  table. Initialization can select an existing source row key for the
  prediction table. It uses explicit single-writer admission with serialized
  score runs, while Core retains shared admission for multi-writer deployments. The
  earlier three-job personal rehearsal trained a Polars model, wrote 600
  initial and 50 later predictions, and replayed without a new commit.
  An optional monthly_paused mode adds a paused monthly schedule to the same
  train job, pins a UTC label window and Delta version, and compares each new
  candidate with the current champion. The default keeps promotion and scoring
  selection manual. Optional automatic champion mode requires a heldout metric,
  minimum improvement and absolute quality threshold. It uses checked alias
  receipts and the same serialized score job, which pins champion per run.
  Bundle initialization now selects whether a new pinned model version keeps
  prior predictions and appends only new rows, or scores a new physical
  generation and switches a stable view after success. Optional retraining
  cron and timezone are editable Bundle variables.
  The live Polars/serverless rehearsal verified first champion, improved-model
  promotion, tied-candidate rejection, both scoring policies, failed-score
  recovery, queued no-op requests and a restricted-principal alias-write denial.

• Prediction Delta targets use run_id, model_name and model_version
  metadata columns. Existing test targets with the former __skyulf_*
  schema require a new target; the writer does not migrate tables.

Fixed

• Spark identifier validation tolerates Databricks serverless hiding
  spark.sql.caseSensitive, retaining conservative collision checks while
  propagating unrelated errors. The failure was reproduced in a live job;
  the correction passed both inference modes in a real serverless Spark Connect
  4.2.0 job using a Polars-trained model loaded from Unity Catalog.
• Delta batch/admission tolerates the explicit serverless restrictions on
  REFRESH TABLE and prediction caching. Identity/ownership reads and
  transactional checks remain active; uncached execution stays distributed.
  Other runtime failures propagate, and classic Spark still refreshes/caches.

Added

• A custom local-engine Databricks Bundle template generates two default
  jobs for candidate training and automatic incremental scoring, using
  existing Skyulf Core services and an editable pandas/Polars
  configuration. The earlier isolated lifecycle rehearsal passed
  train/compare/stage/promote and 2+2-row incremental scoring; the later clean
  generic rehearsal used one source table and proved 600+50-row scoring with a
  no-op replay. Existing predictions retained their pinned model version after
  alias promotion. Spark FE/model execution and endpoints remain separate.

• Explicit MLflow champion promotion and rollback revalidate a pinned local
  model comparison, acquire shared admission, check the expected alias version
  and record durable transition receipts. Local registry tests and an isolated
  Unity Catalog run verified promotion, conflict rejection, rollback and a
  restricted principal's alias-write denial.

• Read-only local MLflow candidate/champion comparison on one bounded labeled holdout
  reports pinned model/data identities, held-out metrics, improvement and a
  directional quality gate. Held-out evaluation now reuses Core's full model
  metric calculators, including probability-based classification metrics when
  defined, rather than maintaining separate formulas. It never moves an alias.

• Label-aware local candidate training reads a pinned bounded Delta snapshot,
  excludes labels unavailable at cutoff, fits pandas or Polars Skyulf pipelines
  before a temporal holdout, logs the model and comparison evidence to one
  MLflow run, and registers a concrete model version without changing aliases.
  A real taxi serverless test compared two UC model versions on 1,770 holdout
  rows and verified six late labels stayed out of both fit and evaluation.

• Local-engine Databricks SDK configuration and preflight for bounded
  pandas/Polars batches. It selects trusted local or portable artifacts,
  resolves MLflow aliases once to concrete versions, checks metadata and
  source/sink compatibility, and can probe the fitted predictor with a sample.
  Existing Skyulf FE nodes and models need no separate SDK allowlist.

• Explicit local-engine training and UC batch scoring: a bounded split produces
  the existing full-pipeline artifact; a versioned Delta source is filtered by
  requested month, narrowed, row-limited and checked against a decoded-byte
  budget before whole-frame pandas/Polars prediction. The result carries source
  and model identity diagnostics.

• Bounded local pandas/Polars predictions can now publish one requested month
  to a precreated UC Delta table through an explicit Spark result bridge.
  The local_pipeline batch mode pins source, model, code and expected target
  versions, reuses guarded period replacement and shared admission, and rejects
  accidental empty or stale writes. Isolated serverless jobs verified 80
  predictions for each of two months, preserved prior rows and replayed without
  another commit. No SQL Connector or Spark FE/model execution is required.

• Automatic incremental local batch scoring reads an initial bounded Delta
  snapshot, then only new inserts from Change Data Feed. It appends keyed
  predictions and the source-version receipt in one guarded Delta commit;
  scheduled runs need no manual dates or source versions. A source timestamp
  is optional, and updates/deletes or duplicate keys fail explicitly.
  Real local Delta tests cover pandas, Polars and retry/no-op behavior. Two
  isolated serverless jobs validated initial 80-row scoring, automatic scoring
  of 80 later inserts, unchanged prior predictions and a no-op replay.

• A separate real NYC taxi proof trained Skyulf preprocessing and a regression
  model on 3,000 trips, logged held-out MAE/RMSE/R2 and the full artifact to
  MLflow, registered UC model version 1, and scored 200 existing plus 100
  later-inserted trips without changing prior predictions. See the
  live report.

• Versioned local pandas/Polars pipeline artifact and MLflow pyfunc packaging
  preserve fitted encoding/binning, model state, input order, class probabilities
  and explicit tuned thresholds without refitting. The fit engine and exact
  runtime requirements are recorded; registry resolution pins the local payload
  digest. The declared execution scope is whole-frame local batch, with HTTP
  row-local and Spark execution left for separate capability validation.

• Shared Delta-table publish admission with conditional ownership updates,
  non-expiring claims and owner-only release. Real Delta tests cover independent
  sessions, concurrent claims and malformed authorities. Live serverless
  Spark Connect 4.2.0 validation covers monthly replacement/replay, exclusive
  admission across jobs and restricted-principal target-write denial.

• Databricks monthly Delta smoke example, tested with a pinned UC regression
  bundle, preserved prior periods, empty replacement and durable replay receipts.
  The SM-16 evidence records worker package hashes, alias pinning and bounded
  synthetic scale observations for the selected runtime.

• Spark/MLflow initiative: architecture, implementation plans, validation gates
  and an ordered task queue.
  Historical readiness reports are grouped with the initiative.

• Experimental Spark test foundation (SM-00): optional spark extra, separate
  requirements-spark.txt, a Java 17/Python 3.12 CI lane and real JVM/Arrow
  worker smoke tests. The dedicated lane fails if PySpark is missing; normal
  core tests can skip the optional runtime. No Spark node capability is enabled.

• Spark development guide with isolated setup,
  a runtime example, troubleshooting and explicit support boundaries.

• Immutable execution options and row-key declarations, plus explicit
  per-node fit/apply capability checks. Unsupported engine/configuration requests
  produce structured errors without executing a node. Existing local pipelines
  remain unchanged; these contracts do not enable Spark nodes yet.

• Optional Spark engine adapter with real dataframe detection and a separate
  distributed metadata/projection protocol. Local conversion and sklearn
  entry points reject Spark inputs without collecting data. Sessions remain
  caller-owned; base imports require neither PySpark nor MLflow. This adds no
  native FE node or model inference capability yet.

• Opt-in keyed Spark FeatureEngineer entry point: validates schema and composite
  keys, excludes keys/target from feature selection, checks all node capabilities
  before execution, and verifies key/target preservation after native steps.
  Spark dispatcher preparation preserves distributed frames; local FE behavior
  remains available. Unknown distributed metrics replace local profiling guesses.
  Node support is enabled separately by native implementations.

• Versioned portable-state codec for StandardScaler and mean/constant
  SimpleImputer artifacts. Tagged UTF-8 JSON preserves scalar types, non-finite
  values and column order, with explicit byte limits, structural validation and
  a semantic checksum. Arbitrary Python objects are rejected; existing pipeline
  persistence and fingerprint behavior remain unchanged.

• Native Spark SimpleImputer mean/constant fit and apply: one aggregate result
  holds learned values and missing counts; apply uses Spark expressions without
  pandas conversion or Python UDFs. Portable state supports cross-engine apply
  with pandas/Polars. The keyed runner normalizes defaults, preserves explicit
  empty selections and enforces configured state-byte limits. Median/mode remain
  unsupported; the Spark guide includes pipeline and cross-engine examples.

• Native Spark StandardScaler with population variance, all four centering/scaling
  options and portable pandas/Polars/Spark state. Two bounded aggregate results
  compute statistics relative to a numeric reference to avoid large-offset mean
  errors; apply uses native Spark expressions. Null/NaN, constant/near-constant,
  empty and single-row inputs have explicit tested behavior. Numeric selection
  and name validation are shared with the native imputer.

• Portable FeatureEngineer.export_state() / from_state() for ordered imputer
  and StandardScaler chains. Versioned JSON freezes learned column selection,
  validates the total byte budget and rebinds caller-supplied Spark keys/runtime.
  Full-chain tests cover all nine pandas/Polars/Spark fit/apply combinations,
  persistence, repartitioning and bounded driver collection. A runnable Spark
  example and guide demonstrate saving and loading the fitted chain. Successful
  tuned pipelines can export their supported FE; failed model fits invalidate it.

• Standalone inference bundles with explicit raw/features inputs, frozen
  FE/model payloads, ordered input/output schemas, runtime requirements and
  corruption/budget checks. Local pandas/Polars regression and classification
  preserve probability class order and default versus opt-in threshold behavior.
  Successful pipeline fits capture schema metadata without retaining input frames;
  legacy artifacts without schemas require refitting. Includes a runnable example
  and bundle guide. Trusted sklearn
  pickle payloads remain distinct from portable FE JSON and legacy persistence.

• Distributed regression inference through predict_spark(mode="native_features"):
  saved FE runs as native Spark expressions, then the fitted Python estimator runs
  in worker batches. Explicit row keys preserve result identity; schema/order and
  package checks precede submission. Model loading is reused within each iterator,
  and unused input columns do not reach workers. The bundle guide includes a Spark
  example and explains compatibility with existing pipeline/backend artifacts.

• Distributed classification inference through both native_features and
  python_pipeline: string binary and integer multiclass labels retain their
  manifest class order, probability columns use explicit Spark output types, and
  saved tuning/pipeline thresholds match local decisions. Partitioning and Arrow
  batch changes are covered by action-based parity tests; iterator model loading
  is bounded and incompatible worker dependencies fail before unpickling. The
  runnable spark_batch_inference.py example exercises both modes.

• Illustrated inference guide covering
  fitted FE/model reuse, a numerical local/Spark example, and available inference
  paths. Includes four SVG diagrams and editable Mermaid sources.

• Optional MLflow tracking through TrackingConfig and track_run, with a
  dependency-free disabled mode, explicit client-bound run IDs, FINISHED/FAILED
  lifecycle, raise/warn failure policy, and opt-in config artifact digest.
  Install it with skyulf-core[mlflow] or requirements-mlflow.txt.

• Optional MLflow pyfunc packaging for trusted inference bundles. Explicit
  run IDs and client-bound artifact upload preserve run isolation; raw/features
  prediction parity, class/threshold metadata, named signatures, synthetic
  input examples, and clean wheel/subprocess loading are covered. Registry,
  Unity Catalog, Spark UDF and endpoint delivery remain separate gates.

• Optional MLflow registry adapter with explicit run-artifact publication and
  alias-or-version resolution. Alias selection is pinned to a concrete version
  URI, signatures and bundle digests are exposed, tracking and registry stores
  may be separate, and missing/access/dependency failures are typed. Live Unity
  Catalog and Databricks validation remain in SM-16.

• Experimental monthly Spark batch inference with pinned Delta source versions,
  explicit UTC periods and snapshot availability cutoffs, and model/code/run
  provenance in committed receipts. Atomic period replacement preserves other
  periods; retries cannot undo a newer run, stale writers fail, and empty-period
  deletion requires allow_empty=True. Real local Delta tests cover both
  inference modes and process-shared admission. Distributed admission and live
  Databricks validation remain SM-16. Includes an English guide and Delta CI lane.

• Pinned MLflow registry bundles can now be loaded directly for existing local
  or Spark prediction. The loader validates the declared artifact path and
  bundle identity without following a moved alias. A preliminary platform smoke
  script compares a registered pandas/Polars-trained model through both Spark
  modes; full live Databricks validation remains open.

Planned scope — not yet available

• Further native Spark nodes and broader portable-state coverage.
• Worker wheel packaging and scale/RSS validation for production Spark clusters.
• Live Unity Catalog and Databricks registry validation.
• Databricks scheduling and distributed publish admission for batch jobs.

Endpoint integrations and template/Databricks Bundle generation follow the
core and batch validation gates. Planned items are not shipped functionality;
verified delivery notes will be added as the tasks are completed.