Skip to content

v0.9.4

Choose a tag to compare

@github-actions github-actions released this 18 Oct 17:01
· 133 commits to master since this release
1f7cc56
  • Configure scoring eligibility, prediction bands and combined model outputs in
    config/scoring.yml, with saved custom Python helpers where needed. New Bundle
    projects use YAML; existing Python-only policies keep their behavior.

  • Warn about the new learned preprocessing steps before a train/test split even
    while the Canvas registry request is pending or unavailable.

  • Reject invalid serving-monitor enrollment batches before acquiring the shared
    reference lock, so configuration errors cannot write a lock record or be
    hidden by an unavailable monitoring store. Valid enrollments remain guarded.

  • Add FrequencyEncoder, RareCategoryGrouping, QuantileTransformer, Normalizer,
    IsolationForest and LocalOutlierFactor with saved-state checks, Canvas controls
    and weighted-pipeline row alignment. Joint outlier inference preserves request
    rows with exclusion reasons; LOF distinguishes training from novelty scoring.

  • Add fit-time regex/dtype/exclusion column selection, hierarchical multi-column
    GroupImputer fallback, learned group deviations, cyclic/local-time date
    features, independent Count/TF-IDF vocabularies per column, per-column
    Winsorize percentiles, RFECV and sequential feature selection. Expose target
    encoder CV/shuffle/seed, MinMax clipping and Robust unit-variance options.

  • Reject misspelled preprocessing choices, malformed ordinal orders and invalid
    hash widths instead of silently using fallbacks. Standardize new HashEncoder
    defaults on eight buckets, exclude targets from automatic polynomial expansion,
    and normalize missing TargetEncoder category identities for new fits.

  • Validate simple-transform rule containers before reading their entries, so
    malformed rules fail clearly and rejected iterators remain unconsumed.

  • Keep floating sentinel-to-null replacements numeric. Interpret constant
    imputation text in each selected column's numeric/Boolean type with explicit
    errors for invalid or lossy values. Improve schema previews without guessing
    engine- or data-dependent output types.

  • Version new FeatureMath operation validation and nullable calendar outputs;
    previously saved unversioned operations retain their replay behavior. Keep
    existing arithmetic null/zero rules and exact integer clipping boundaries.

  • Document tested built-in alternatives beside custom Bundle teaching examples,
    including differences in missing values, output names and dtype handling.

  • Add explicit ordered string categories and optional column_names: slug to
    OneHotEncoder, including Canvas controls. Fixed vocabularies produce stable
    columns even when a training split lacks a category; legacy saved encoders
    retain their original names and behavior.

  • Compute context-dependent SHAP preprocessing before sampling explanation rows,
    preserving lag/rolling history and exclusion counts. Row-independent pipelines
    retain early sampling and the same SHAP budgets.

  • Raise an explicit error when a saved GeneralTransformation power transform
    cannot apply, instead of silently substituting raw values for the whole column.
    Previously saved pipelines that relied on that fallback now fail explicitly.

  • Return an unknown preview schema for GroupImputer automatic numeric selection,
    whose selected columns depend on observed binary/constant values; explicit
    column selection is unchanged.

  • Raise the SHAP explainability.max_features guard from 50 to 100 so wider
    models, such as fixed one-hot feature sets, can be explained.

  • Add optional initial_scoring: from_now: the first batch records a Delta
    checkpoint without predictions, subsequent batches score new inserts, and
    model changes/recovery preserve the original boundary. full_snapshot
    remains the default. Normal rebuilds reuse saved prediction keys and recent
    CDF; CDF-expiry recovery requires the original starting snapshot.

  • Generate config/sources.sql only when setup selects include_source_sql;
    the source guide and read-only source checks remain available in every project.

  • Keep outcome availability columns on the outcome table in the optional
    config/sources.sql starter for separate_outcomes, so measurement tables
    do not require label metadata for single-model, competition or model-set training.

  • Generate source layouts, outcome-table bindings and task-aware performance
    policies from setup answers. Quick setup includes table arrangement; generated
    SOURCES.md, SQL starters and preview.py --action sources explain and check
    existing source contracts without creating source tables.

  • Train from an optional separate outcome table using exact record keys and
    pinned Delta versions/identities. Newly available labels can enter guarded
    retraining even when feature rows have not changed. Saved runs retain the
    original outcome binding; embedded-target configurations keep their identity.

  • Expose sampling and preprocessing diagnostics in detailed setup, explain row
    and byte limit failures, and warn about initial in-sample scoring and scheduled
    fixed training windows. Monitoring jobs accept native email/webhook and
    duration-warning settings; drift alerts remain separate SQL alerts.

  • Default configured monitoring to the shared skyulf_monitoring schema while
    preserving custom destinations. Generated projects add a separate weekly
    reference cleanup job with confirmation enabled and 90-day retention; personal
    development targets remain paused. Manual cleanup defaults to a dry run.

  • Serialize reference publication, enrollment and generated model activations
    with cleanup through the existing Serializable request table. Failed protected
    writes retain their owner; validation failures before a write release it.
    Known contention waits within a bounded deadline. Explicit recovery verifies
    job identity/termination and audits the release. All writers sharing a store
    must participate or be paused.

  • Write format-2 monitoring references as one population with training membership
    plus a receipt; share source tables only when verified content and schema match.
    Legacy references remain readable, and existing tables are not migrated.

  • Monitoring can treat per-column missing-value markers, such as a -200
    sensor reading, as missing: add missing_markers to
    monitoring_drift_thresholds. Markers compare exactly in the column's own
    integer or floating-point type. Reports without the key are unchanged.

  • Add a single-table air-quality walkthrough: train on summer observations,
    append winter days with separate outcomes, and score new rows with Spark.
    Serial loader retries recover missing inputs or labels without duplicate
    inserts. Use a shared monitoring schema for the air-quality examples and
    clarify that daily performance windows follow prediction time, while
    report mode records the first eligible loss without waiting for a streak.

  • Pin sqlalchemy==2.0.43 in generated deployment/requirements.txt. MLflow
    3.16.1 allows any sqlalchemy<3, and 2.1.5 has no wheel for the serverless
    aarch64 image, so its source build failed and stopped every job at library
    installation.

  • Add the prepared real-data air_quality_no2 example on UCI Air Quality:
    built-in and custom pre-split filters, custom marker/category/fitted steps
    with built-in IQR exclusion, Yeo-Johnson, imputation, one-hot and scaling,
    combined scoring rules, Spark UDF scoring, SHAP, evaluation charts and
    monitoring on a later season with a report-only WAPE policy. The real-data
    loader gains the pinned archive and a datasets widget for selective loading.

  • Add unweighted WAPE alongside existing regression metrics, with an explicit
    unavailable outcome for all-zero actual values. Support heldout_wape in
    model comparison/quality gates and wape in performance policies; show its
    label, explanation and lower-is-better direction in Kayra.

  • Configure categorical grouping, uniqueness, summary caps, PSI severity bands
    and non-Spark monitoring row/byte budgets through the existing
    monitoring_drift_thresholds JSON variable. Preserve absent-key defaults and
    reject invalid values before measurement; Spark and frame engines share validation.

  • Measure repeated categorical drift above 50 reference values in pandas/Polars
    and Spark using the top 50 reference categories plus a pooled tail. Bound
    domains at 10,000 categories and skip reference uniqueness above 0.5. Preserve
    existing small-domain results and add a diagnostic new-category rate on the
    grouped path; statistical support is still required for drift alerts.

  • Prefer build_model_rules() in project feature packages while retaining
    build_scoring() for older projects. Generated projects no longer need an
    alias. Correct automatic admission of reused pre-split rules and explain
    row-only replay, exclusions and admission settings in the template.

  • Add the prepared taxi_fare_robust example: fixed eligibility, fitted IQR,
    Yeo-Johnson, imputation, encoding and scaling, with Spark UDF scoring and an
    explicit raw-input endpoint for REST and typed SQL ai_query requests.

  • Explain sampled rows accepted by fitted exclusion steps with SHAP, reporting
    sampled, accepted and excluded counts and reasons in JSON and HTML. Fewer
    than two accepted rows remains unavailable; samples are not topped up.

  • Generate evaluation charts for models with excluded predictions. Preserve
    the saved holdout sample, plot only its predicted rows, and disclose sample
    counts in captions and task output. Invalid predicted values still fail.

  • Configure new serving endpoints through online_serving.endpoint in
    config/serving.yml: CPU workload type, Small/Medium/Large size, scale to
    zero and a positive readiness timeout. Defaults remain CPU/Small/true,
    with a 3600-second setup wait. Setup returns READY only after the exact
    configuration settles; timeout leaves the endpoint available for inspection.
    Existing endpoints are never overwritten. Rollout requires matching compute
    settings for champion and challenger; older default plans remain readable.

  • Compare floating probe outputs with fixed relative tolerance 1e-9 and
    absolute tolerance 1e-12. Labels (including tuned labels), integer values,
    schema, order, null positions and input/state mutation checks remain exact.
    Version 2 evidence records this policy; regenerate version 1 evidence by
    retraining and repackaging the model.
    Repeating an identical request still requires exact outputs. Admission also
    checks the complete scoring result, including custom outputs and model-set
    composition, so tolerated intermediate noise cannot hide a changed final
    decision. Each empirical step and final prediction path must actually see
    at least 16 rows; an excluded sample cannot supply empty passing evidence.

  • Preserve requested rows excluded by fitted IQR, ZScore, EllipticEnvelope
    and ManualBounds preprocessing in saved local, Spark and endpoint scoring.
    These rows receive null predictions, scoring_status="excluded" and
    exclusion_reason="outlier:<step name>". Training still removes outliers;
    the raw array pipeline.predict() API retains its row-preservation guard.
    Holdout reports include total, scored and excluded counts. Candidate/champion
    comparison scores each full holdout once, then compares metrics on the rows
    both models predicted. Independent coverage gates default to at least 90%
    scored by the candidate and at most five percentage points lost versus the
    champion. Set min_scored_fraction and max_coverage_drop in generated
    training settings; either accepts null to disable that rule. Fresh promotion,
    manual approval, model-set validation and rollout preserve these settings.

  • Exclude PowerTransformer rows with invalid inputs or non-finite transformed
    values. Training removes those rows with aligned targets and weights and
    reports the counts; saved scoring preserves them with null predictions and
    exclusion_reason="invalid:<step name>". Valid neighbours keep their fitted
    transformation. Missing NaN/null values pass through for a following imputer;
    they are not invalid-input exclusions. Training stops if invalid rows exceed
    10% of that step's training input; exactly 10% remains allowed. Missing values
    do not contribute to the invalid count. Use Yeo-Johnson for values containing
    zeros or negatives. All-invalid training and invalid learned scaler state raise
    clear errors. Existing Delta prediction tables must match the added status
    columns; use a new or explicitly migrated table rather than implicit schema
    evolution. Monitoring exposes total and invalid-input exclusion rates in the
    existing data quality table, while performance uses predicted rows only.
    Deterministic IterativeImputer replay
    now runs its saved rounds for all-missing requests too; posterior sampling
    and unsupported estimators remain excluded from partition admission.

  • Package the exact XGBoost/LightGBM versions retained by fitted models,
    including ensemble members and saved preprocessing state. Reuse the optional
    dependency checks before unpickling; unused native packages are not required.
    Re-export older native artifacts that lack these pins. Honor the existing
    Python major/minor compatibility policy when rechecking a saved manifest,
    allowing patch-version differences while rejecting changed dependency pins.

  • Bind XGBoost and LightGBM semantic fingerprints to their serialized learned
    trees, including LightGBM's prediction iteration limit. Reject unknown ctypes
    state and native booster overrides. Earlier native-model fingerprints and
    partition evidence must be regenerated and models packaged again; old evidence
    is not silently accepted under the corrected format.

  • Admit additional exact built-in models and voting/stacking on Spark workers
    and serving endpoints when training seals their partition probe evidence.
    Check nested estimator identities, saved state, predictions, probabilities
    and configured thresholds. Eligible row-preserving preprocessing recipes
    and four lightweight text steps use the same bounded evidence mechanism;
    unsupported row filters and global/window transformations remain rejected.
    Missing or changed evidence requires retraining. Built-in pipelines do not
    need a project source directory. Validated empty saved-model requests return
    the declared empty schema without invoking the estimator.

  • Admit pandas SimpleImputer(median) and GroupImputer(median) on the saved
    pipeline worker path using their learned values. Native Spark/portable
    imputer codecs keep their existing supported strategies.

  • Add two prepared real-data projects: a Wisconsin Diagnostic classifier and a
    Bike Sharing model set with separate casual and registered demand predictions.
    Both demonstrate custom preprocessing, Spark UDF scoring, serving endpoints,
    SHAP, dataset Inputs and monitoring. The examples include data loaders and
    Spark/REST parity checks and are regenerated from the Bundle template.

  • Preserve temporal and group CV metadata during custom-step admission, while
    rejecting altered metadata-removal records. Local admission probes now use
    the workflow's CV settings, and model-set packaging inherits its custom-step
    admission setting.

  • Plot nested CV's saved outer-fold scores and keep custom outputs such as
    probability_band separate from class-probability columns in monitoring.

  • Let Spark scoring and endpoints run your own column_step/fitted_step
    steps and saved eligibility and output scoring rules. Projects with
    inference_mode: spark or online_features check them automatically during
    training; other projects set admit_probed_custom_steps: true. Training lints
    the source for whole-batch calls and writes to shared process state, then
    probes each step and rule on 16 to 256 holdout rows (whole, repeated, chunks
    of 1, 2 and 7, reversed, renumbered index, empty, one row's inputs changed,
    and a freshly loaded copy of the project code) and saves passing evidence
    inside the model. Model-set combined function rules from
    build_combined_rules() are probed the same way when the set is packaged and
    admitted with their sealed evidence. Reused pre-split eligibility
    (SCORING_MODE pre_split or combined) is admitted when its steps are
    row-local and probed; Deduplicate and custom filter classes in the reuse stay
    rejected. inference_context="row" is optional; group, window and
    global steps stay rejected. Spark output reserves the text ␀null␀ for
    nulls and rejects a real value equal to it. A failure stops training before registration and
    names the step or rule, the check, the file and line for lint findings and
    one fix. preview.py --action train runs the lint offline and
    smoke.py --probe <sample> runs the probe on a local sample. Spark output now
    carries rule nulls (exclusion_reason, excluded predictions) to the driver.
    The Spark pyfunc parameters are renamed spark_output and spark_batch_rows
    (formerly skyulf_spark_output and skyulf_spark_batch_rows). Packages logged
    by an earlier release already fail the runtime source check, so log them again.
    See the
    opt-in section.

  • Preserve whitespace inside MLflow alias-recovery evidence while keeping each
    Unity Catalog tag free of boundary whitespace. Read existing intent records;
    upgrade lifecycle jobs sharing a model together because older readers cannot
    recover the new framed records. Run recovery with this release as well. The
    framing reduces the bounded intent capacity from 12,288 to 12,192 characters.

  • Start the final training report with a Where to look table: one row per
    fitted model linking its MLflow run, registered version, training data counts,
    preprocessing diagnostics, SHAP report, evaluation charts and saved decision
    thresholds. It reads saved evidence only and never blocks the lifecycle result.

  • Show model-set approve, reject and rollback values as the same copyable
    Available action and If rollback is needed tables as single models.

  • Ask for the monitoring store catalog and schema in quick and detailed setup.
    Blank answers generate monitoring_enabled: "false", so a new project trains
    and scores before a monitoring destination exists; previously monitoring was
    enabled with an empty destination and both jobs failed until it was set.

  • Advertise the parameter names preprocessing nodes actually read in registry
    metadata: alias_type, column_types, correlation_method, label_format
    and transformations replace the ignored domain, type_map, method,
    encode and func defaults.

  • Let training continue when the champion was produced by another Core release:
    the comparison reports champion_runtime_unavailable, automatic promotion holds,
    and an operator can approve a reviewed candidate that passes its absolute quality
    gates with the new train-job parameter allow_unloadable_champion=true.
    Previously the first training after an upgrade failed and no candidate could
    replace the champion. Model sets report the limitation explicitly.

  • Ask for the endpoint inference-table mode and name prefix when online serving is
    selected.

  • Grant CAN_MANAGE on every generated target to the deployer and a new
    pipeline_owner variable (CI/CD can pass --var pipeline_owner=<email>), and
    CAN_VIEW on shared targets to an optional per-environment group. Detailed setup
    asks for the owner and the test, syst and prod groups; blank answers use the
    deployer and add no group. Previously every target had permissions: [].

  • Add four prepared NYC taxi example projects (Spark-scored single model, online
    single model with feature lookups, competition, model set) with a SQL loader and
    a feature walkthrough under templates/databricks/examples/nyctaxi, checked
    against the template in CI.

  • The setup wizard no longer asks ensemble voting weights or per-member voting,
    cv, passthrough, calibrate and calibration_cv settings; member models are
    still asked. The settings keep their defaults in config/training.yml, and answer
    files can still set them. Generated projects are unchanged.

  • Refuse at bundle init, or stop offering, setup combinations that generated a
    project failing later: the same ensemble base model twice; halving on
    n_estimators for a model without it; heldout_mape, heldout_precision_weighted
    or heldout_recall_weighted in a competition; stratified CV for regression; a
    full_snapshot window with a temporal split (now asked first, and the split is
    then random); online serving with multi_target or Polars. search_max_resources
    defaults to 100 instead of an unusable null.

  • src/tools/smoke.py now runs the search, competition-metric and
    config/serving.yml checks that previously failed only in preview or in the job.
    preview.py passes the CDF recovery setting for model sets and reports a broken
    configuration in one line instead of a traceback.

  • An unfilled generated feature lookup reports that it is not filled in, and every
    invalid Unity Catalog table name gets one message naming the
    catalog.schema.name form.

  • Online feature checks name what failed: the empty features, how many rows and the
    first row for a missing lookup value, and the age against max_age_seconds for a
    stale feature. Serving endpoints show these only in the served model's logs, because
    the native Feature Engineering wrapper returns a generic error to callers.

  • Fix approving a candidate with allow_unloadable_champion=true under automatic
    promotion: the job's evidence lookup accepted only manual_approval.

  • Add logging_mode: disabled for serving endpoints whose callers store their own
    predictions, for example through ai_query. Rollout and serving monitoring
    reject such endpoints because they read payload logs.

  • Record the Bundle target that ran the daily serving rollout in its saved
    observation, instead of the fixed rollout.monitoring.environment: syst label.

  • Remove src/features/custom/ from new projects. Scoring callbacks live at the
    end of src/features/scoring.py and are selected as scoring.<function>; own
    factories go in preprocessing.py or a module under src/features/. Existing
    custom.<file>.<factory> references keep working.

  • Reorganize integration documentation around an ordered Databricks project
    walkthrough, with dedicated training, scoring, feature-table, troubleshooting
    and glossary references. Keep generated project commands offline; shorten its
    root README and separate training/scoring topics. Document every registered
    preprocessing operation and accepted training YAML field, with executable
    examples and CI checks for catalog coverage, diagnostics and public links.
    Give model sets, monitoring, project removal, custom pre-split filters, Spark
    scoring, ensembles and Spark benchmarks their own pages; split the preprocessing
    catalog into family pages; group the Databricks navigation by task; add a
    troubleshooting message index, glossary terms and flow diagrams.

  • Keep training field tables in one reference and link the generated project
    guide to it. Add model/search-space and tuning references, plus executable
    custom preprocessing and pre-split examples with their YAML selectors and
    scoring eligibility rules.

  • Document reference preparation for a first pinned candidate's monitoring
    handoff, including safe score replay after an already committed prediction.

  • Add saved training candidates, selected-model parameters and dataset provenance
    to Training and lineage, using the shared model/version/context filters.
    Capture bounded optional summaries in new monitoring observations; preserve
    missing evidence, candidate/run ownership and immutable observation identity.

  • Remove the Execution and compute monitoring page and its six exclusive
    datasets. The dashboard no longer queries job, compute or billing system
    tables; inspect those details directly in Databricks.

  • Share monitoring observation, identity and serving-window SQL through three
    owned views. Preserve dashboard metrics, filters and chart presentation. Store
    owners must initialize the updated views and grant viewer access before
    upgrading producers and publishing the dashboard.

  • Fix clipped monitoring page titles and pin daily drift/performance problem
    series to red. Render serving error classes and execution times as separate,
    labelled bars while preserving missing measurements. Show PSI limit crossings
    in a heatmap with distinct identity labels; exact PSI, limits and ratios remain
    available in tooltips and the existing detail table.

  • Batch Spark drift population checks, finite counts, minima and centered
    reference standard deviations across features. Preserve numeric-string
    conversion, missing-data precedence and native integer precision; exact CDF
    and bounded histogram calculations remain per feature.

  • Batch serving-payload validation into one Spark aggregate per phase, including
    typed input and output columns, instead of scanning once per check or feature.
    Preserve validation order, error messages, empty-window behavior and null values.

  • Apply rollout maximum_latency_ms to p99 model execution time when recorded,
    falling back to maximum latency only when p99 is absent or null. This changes
    the previous maximum-only policy; review configured limits before upgrading.
    Invalid non-null p99 holds rollout; other admission and promotion checks remain.

  • Keep serving monitoring usable when individual requests have logging errors.
    Retain deduplicated HTTP counts, exclude flagged payloads from feature and
    performance metrics, and show the affected request count in saved JSON and
    notebook notes. Mark otherwise healthy reports degraded while preserving
    measured drift and no-data status. Drift and performance on usable payloads
    remain eligible for retraining under the existing data and policy checks;
    performance-policy output exposes its own window's excluded request count.
    Keep sampling/identity validation strict; partial capture holds live rollout.
    Historical reports are unchanged; recomputed summaries receive new identities.

  • Show requests with logging errors in the existing Online serving snapshot and
    history tables and the captured-request chart tooltip. Preserve recorded zero
    and show Not recorded for absent counts, without changing HTTP error classes
    or chart series.

  • Fix two Polars 2 failures that earlier Polars versions did not show. StandardScaler
    no longer crashes on unsigned 64-bit columns with integer-typed saved means, and model
    matrices mixing UInt64 with signed integer columns convert to float64 as before.
    Both cases used an Int128 intermediate that has no NumPy conversion.

  • Add explicit recovery for interrupted MLflow alias operations. Persist a
    bounded intent before changes; complete fully applied roles or abort untouched
    state under shared admission, without moving aliases. Refuse partial or legacy
    evidence, preserve pending ownership, and verify terminal receipts before
    clearing the guard. Document manual recovery and Unity Catalog tag capacity.

  • Timestamp lifecycle event tags in UTC; continue reading older receipts and preserve
    replay behavior. New tags require Core 0.9.4 or newer: upgrade all jobs sharing a
    model's lifecycle tags together; older readers and downgrades reject timestamped
    events. Reserve space for the final state before alias changes begin.

  • Preserve unmeasured monitoring counts as SQL NULL instead of fabricated zeros.
    Explicit zero counts remain zero; saved history and retry identities are unchanged.

  • Advance single-model and model-set incremental watermarks through empty source
    windows with an atomic zero-row Delta receipt. Preserve temporal history and
    same-version retries; monitoring skips the empty batch but retains delayed-label
    checks.

  • Allow rollback to restore a valid prior champion when its older previous-champion
    version was deleted. Clear that alias, retain the null pointer in the receipt,
    and verify it on retries; permission and artifact errors still prevent changes.

  • Retain explicitly excluded prediction counts in full Spark, pandas and Polars
    monitoring reports and their saved JSON, separately from missing predictions.

  • Retain the configured performance-policy window count in direct evaluations
    and failed monitoring observations. Notebook reports mark missing historical
    counts as unavailable instead of showing a misleading 0 / 0 failure streak.
    Equivalent old/new history replays retain their streak across the added field.

  • Provision the retraining-request table during central monitoring setup with
    ownership/schema checks and Serializable isolation. Automatic training requests
    reuse pre-provisioned storage without requiring table-creation privileges.
    Before upgrading producers without creation privileges, have the store owner
    rerun initialize_monitoring_store to add the table to older monitoring stores.

  • Check pending-event ownership before clearing an MLflow alias transition.
    Preserve another writer's marker and report an uncertain outcome if ownership
    changes or the marker disappears before completion.

  • Validate both rollback destination versions before changing MLflow aliases or
    receipts. A missing prior champion or unreadable existing destination leaves
    registry state unchanged instead of starting a partial rollback.

  • Align Spark PSI/KL with Core for numeric features with at most ten distinct
    reference values. Preserve discrete levels, fractional shifts and precise
    integer/float boundaries; retain quantile bins for continuous features.

  • Fix incremental single-model and model-set scoring after Delta maintenance.
    Read the last prediction receipt across maintenance commits, retain physical
    version checks for concurrent writes, and report the actual prediction commit.
    Incremental latest-version lookups request one history entry; historical receipt and
    monitoring reads retain their full history.

  • Breaking: Remove the 88 remaining Databricks/MLflow flat-module aliases
    and both _compat packages. Update imports and string-based patch targets to
    the responsibility packages documented in the integration READMEs. Existing
    package-root exports remain available. Old model pickles naming removed paths
    can fail before inner runtime validation; package and validate replacement
    models with current code before upgrading jobs that consume those artifacts.

  • Add captured HTTP status classes, approximate latency percentiles and fixed
    duration buckets to serving observations. Show error classes, tail latency and
    per-window distributions in the monitoring dashboard; preserve missing legacy
    evidence and existing rollout gates. Recomputed windows receive new report IDs.

  • Clarify serving charts with grouped HTTP error classes, distinct legend colors,
    mean/p95 latency bars, and explicit "Not recorded" labels for missing telemetry.

  • Add a quick setup choice to the Databricks project wizard. quick asks only for
    the layout, project name, task, source table, record keys and target/input
    columns and accepts every other documented default. Quick setup is the default;
    choose detailed to review model, search, branch and deployment settings now.
    Explicit answer-file values are honored in either mode.

  • Fix the interactive Databricks project wizard skipping questions. The CLI
    evaluates only properties, required and anyOf in a skip rule and silently
    ignores not and allOf, so the wizard never asked for the branch count,
    per-branch settings, competition candidates, search strategy or ensemble
    members. The schema builder now rewrites every rule into the supported form
    and fails on a rule the CLI cannot evaluate. Detailed setup asks the questions
    it describes; quick setup also asks each target's task and columns. Lists start
    with their default, so Enter selects it. Answer-file initialization is unchanged.

  • Name the setting in the feature lookup table error raised for an unfilled
    placeholder: feature_lookup.lookups[N].table_name.

Upgrade steps for generated Databricks projects

A generated project records its release in deployment/requirements.txt and in
the first lines of its README. Before raising that pin from 0.9.3, in order:

  1. Package and validate replacement models with this release before upgrading
    jobs that load saved models. Models pickled against the removed flat-module
    import paths can fail to load.
  2. As the monitoring store owner, run initialize_monitoring_store once. It adds
    the retraining-request table and the complete_observations,
    monitoring_identities and serving_windows views, and refreshes
    model_version_details with training_summary_json. Producers without
    creation privileges fail until this has run.
  3. Grant dashboard viewers SELECT on the three new views.
  4. Review each rollout maximum_latency_ms. It now limits p99 execution time
    when recorded, instead of the maximum.
  5. Raise the pin and redeploy, in one change, every job that shares a model's
    lifecycle tags. Older readers and downgrades reject the new timestamped events.
  6. The dashboard definition is a file copied into the project. To adopt the new
    one, replace src/monitoring/monitoring.lvdash.json with the file from a
    project generated with this release, after step 2, then deploy. It no longer
    has the Execution and compute page.
  7. Optional, for links in the final report's Where to look table: add
    workspace_host: ${workspace.host} to the training_report task parameters in
    resources/train.job.yml, and display_html=globals().get("displayHTML") to
    the call in src/jobs/training_report.py (models_report.py for model sets),
    as in a newly generated project. Without them the report still runs; links
    appear as plain text.
  8. Add the allow_unloadable_champion train-job parameter (default empty) and pass it
    to the lifecycle tasks, as in a newly generated project, so a champion from the
    previous release can be replaced.
  9. Optional: add bundle_target: ${bundle.target} to the daily_rollout task
    parameters in resources/daily_rollout.job.yml so rollout evidence names the
    target that ran it.
  10. Optional, when your CI/CD passes --var pipeline_owner or expects target
    permissions: add a pipeline_owner variable to deployment/variables.yml and the
    permissions list to each target in deployment/targets.yml, as in a newly
    generated project. Undefined variables passed with --var fail validation.