Feat!: forward-only changes schema modification check

[treysp]

Some forward-only changes require schema modifications that drop existing data (e.g., changing a column type from str to int). If a user inadvertently makes such a change, they will lose existing data. This feature identifies destructive changes and optionally warns/errors before they occur to avoid accidental data loss.

The feature introduces a new model kind property on_destructive_change with valid values allow, warn, and error (default is error). The on_destructive_change property only applies to incremental models and can be specified as a model default.

Users may want to temporarily allow a destructive change to a forward-only model. The feature introduces to the plan command a new --allow-destructive-model selector to temporarily allow destructive changes.

Of note

• Defaults

  • on_destructive_change is a property of a model kind
    • Only incremental models are checked for destructive changes, and their default is ERROR
    • Non-incremental models default is ALLOW
  • Users may specify a default in the model_defaults configuration key - it will only apply to incremental models, as non-incremental models are always allowed

• Plan time check

  • Uses SchemaDiffer and engine-specific rules to determine whether a change is destructive.
  • We pass the current engine's SchemaDiffer into the plan builder to perform the comparison.
  • Evaluating a change requires that columns_to_types is known.
  • Evaluates both direct and indirect changes to a model. A complete evaluation requires knowing columns_to_types for the model and all directly modified ancestors.

• Run time check

  • Uses the set of Alters passed from SchemaDiffer to alter_table() to determine whether a change is destructive.

• dbt

  • We map dbt's incremental model on_schema_change setting to sqlmesh on_destructive_change
  • on_schema_change values are mapped to on_destructive_change as follows:
    • ignore -> WARN
    • append_new_columns -> WARN
    • fail -> ERROR
    • sync_all_columns -> ALLOW