Move column nullability and DEFAULT ownership to convergence

Removes the schema editor's 4-way backfill for nullable -> NOT NULL
transitions and delegates nullability + column DEFAULT reconciliation
entirely to convergence, which already handles them with online-safe
DDL (CHECK NOT VALID -> VALIDATE -> SET NOT NULL; catalog-only
SET/DROP DEFAULT).

- Add allow_null to the previously named non_db_attrs so the schema
  editor short-circuits nullability-only alters, same pattern that
  already covered default=.
- Rename non_db_attrs -> non_migration_attrs. Post-refactor the
  attribute covers allow_null, default, on_delete — all real DB
  concepts, just not owned by migrations. "non_db" was a Django-ism
  that no longer matches Plain's architecture.
- Strip non_migration_attrs in the autodetector before deciding to
  emit AlterField, so migration files no longer carry no-op operations
  for nullability/default/on_delete/choices-only diffs. These were
  previously generated and silently short-circuited at apply time.
- Delete the autodetector's "alter nullable -> NOT NULL without
  default" MigrationSchemaError. Convergence reports NullabilityDrift
  with has_null_rows=True as a blocked fix and tells the user to
  backfill; no longer a structural failure.
- Delete unused schema editor helpers (_alter_column_null_sql,
  _alter_column_default_sql), SQL templates (sql_alter_column_null /
  _not_null / _default, sql_update_with_default / _db_default), and
  the dead sql_alter_column_type class attr. Clean up the unused
  new_type param on _rename_field_sql.
- Simplify _alter_column_type_sql: drops the always-[] post_actions
  tuple and the model / old_field parameters it never used; uses a
  local variable instead of mutating self.sql_alter_column_type on
  each call.
- README: add "How do I make an existing column NOT NULL?" FAQ.
  Update the add-a-field FAQ to point at the convergence-driven flow
  (add allow_null=True, data-migrate, remove allow_null=True). Drop
  the 4-way backfill example from the DDL-timeouts paragraph.

Upgrade path: projects upgrading will keep any historical AlterField
operations from past allow_null flips. These re-apply via the schema
editor, which now no-ops on those diffs, so migrate apply is safe.
The first postgres sync after upgrade will surface NullabilityDrift
for any column whose DB state disagrees with the model — intended
behavior; convergence applies SetNotNullFix automatically if no NULL
rows exist, or blocks with guidance if they do.

Author: davegaeddert

plain-postgres/plain/postgres/README.md

@@ -693,7 +693,7 @@ POSTGRES_CONVERGENCE_STATEMENT_TIMEOUT = "3s"
 
 `lock_timeout` applies to every DDL. `statement_timeout` applies only to statements that take `ACCESS EXCLUSIVE` — non-blocking operations (`CREATE INDEX CONCURRENTLY`, `VALIDATE CONSTRAINT`) run unbounded because they can't cascade the lock queue.
 
-If a migration issues a row-touching UPDATE (e.g. the 4-way backfill that runs when you flip a nullable column to `NOT NULL` with a persistent default), the 3s `statement_timeout` will kill it on any non-tiny table. That's intentional — the right fix is a batched data migration, not a hidden inline backfill. The common first-time failure mode is applying migrations against a pre-seeded dev or staging database: raise the ceiling for that one run, then lower it back for production deploys.
+If a migration issues a row-touching UPDATE (e.g. a hand-written `RunPython` or `RunSQL` backfill), the 3s `statement_timeout` will kill it on any non-tiny table. That's intentional — the right fix is a batched data migration, not a single long-running UPDATE. The common first-time failure mode is applying migrations against a pre-seeded dev or staging database: raise the ceiling for that one run, then lower it back for production deploys.
 
 Use `RunSQL(no_timeout=True)` to opt out for a specific operation:
 
@@ -1222,7 +1222,14 @@ Add the field to your model class, then run `plain migrations create` to create
 If the field is required (no `default=` and not `allow_null=True`), the autodetector refuses to generate the migration, since there's no value to seed existing rows with. You have two options:
 
 1. Declare a `default=` on the field so the new column has a value for existing rows.
-2. Add the field as nullable first, scaffold a data migration with `plain migrations create --empty --name backfill_<field>` to populate existing rows, then alter the field to NOT NULL in a third migration.
+2. Add the field with `allow_null=True`, scaffold a data migration with `plain migrations create --empty --name backfill_<field>` to populate existing rows, then remove `allow_null=True` from the field — convergence applies `NOT NULL` on the next `postgres sync`.
+
+#### How do I make an existing column `NOT NULL`?
+
+Edit the field to remove `allow_null=True`. `plain migrations create` won't detect a schema change — nullability is managed by convergence. Run `plain postgres sync`:
+
+- If the column has no `NULL` rows, convergence applies the change with a non-blocking `CHECK NOT VALID` + `VALIDATE` + `SET NOT NULL` pattern.
+- If `NULL` rows exist, convergence blocks and prints the table and column to backfill. Scaffold a data migration with `plain migrations create --empty --name backfill_<field>`, write the backfill, and run `postgres sync` again.
 
 #### How do I create a unique constraint on multiple fields?
 

plain-postgres/plain/postgres/fields/base.py

@@ -132,7 +132,7 @@ class Field[T](RegisterLookupMixin):
     # Kwargs that don't affect the column definition; the schema editor
     # ignores these when deciding whether an ALTER is needed. Subclasses
     # that introduce additional non-db kwargs extend this tuple.
-    non_db_attrs: tuple[str, ...] = ()
+    non_migration_attrs: tuple[str, ...] = ()
 
     def __init__(self) -> None:
         self.name = None  # Set by set_attributes_from_name
@@ -536,7 +536,12 @@ def value_from_object(self, obj: Model) -> T | None:
 class ColumnField[T](Field[T]):
     """Base for fields backed by a column value (required/allow_null/validators)."""
 
-    non_db_attrs = (*Field.non_db_attrs, "required", "validators")
+    non_migration_attrs = (
+        *Field.non_migration_attrs,
+        "required",
+        "validators",
+        "allow_null",
+    )
 
     def __init__(
         self,
@@ -642,7 +647,7 @@ def deconstruct(self) -> tuple[str | None, str, list[Any], dict[str, Any]]:
 class DefaultableField[T](ColumnField[T]):
     """Base for column-backed fields that accept a user-provided ``default``."""
 
-    non_db_attrs = (*ColumnField.non_db_attrs, "default")
+    non_migration_attrs = (*ColumnField.non_migration_attrs, "default")
 
     def __init__(
         self,
@@ -700,7 +705,7 @@ def deconstruct(self) -> tuple[str | None, str, list[Any], dict[str, Any]]:
 class ChoicesField[T](DefaultableField[T]):
     """Base for fields that accept a ``choices=`` parameter."""
 
-    non_db_attrs = (*DefaultableField.non_db_attrs, "choices")
+    non_migration_attrs = (*DefaultableField.non_migration_attrs, "choices")
 
     def __init__(
         self,

plain-postgres/plain/postgres/fields/related.py

@@ -96,7 +96,11 @@ def lazy_related_operation(
 class RelatedField(FieldCacheMixin, Field):
     """Base class that all relational fields inherit from."""
 
-    non_db_attrs = (*Field.non_db_attrs, "limit_choices_to", "related_query_name")
+    non_migration_attrs = (
+        *Field.non_migration_attrs,
+        "limit_choices_to",
+        "related_query_name",
+    )
 
     # RelatedField always has a remote_field (never None)
     remote_field: ForeignObjectRel
@@ -359,7 +363,11 @@ class ForeignKeyField(ColumnField, RelatedField):
     ForeignKeyField targets the primary key (id) of the remote model.
     """
 
-    non_db_attrs = (*RelatedField.non_db_attrs, *ColumnField.non_db_attrs, "on_delete")
+    non_migration_attrs = (
+        *RelatedField.non_migration_attrs,
+        *ColumnField.non_migration_attrs,
+        "on_delete",
+    )
 
     empty_strings_allowed = False
 

plain-postgres/plain/postgres/migrations/autodetector.py

@@ -27,6 +27,17 @@
     from plain.postgres.migrations.state import ProjectState
 
 
+def _strip_non_migration_attrs(
+    dec: tuple[str, list[Any], dict[str, Any]], non_migration_attrs: tuple[str, ...]
+) -> tuple[str, list[Any], dict[str, Any]]:
+    # Mirrors DatabaseSchemaEditor._field_should_be_altered — the schema editor
+    # already short-circuits when only non_migration_attrs differ; stripping
+    # them here skips the no-op AlterField too, so migration files reflect
+    # real DDL.
+    path, args, kwargs = dec
+    return path, args, {k: v for k, v in kwargs.items() if k not in non_migration_attrs}
+
+
 class MigrationAutodetector:
     """
     Take a pair of ProjectStates and compare them to see what the first would
@@ -804,9 +815,9 @@ def _generate_added_field(
                 f"Choose one:\n"
                 f"  1. Declare a default on the field in models.py, e.g.\n"
                 f'       {field_name}: str = types.TextField(default="...")\n'
-                f"  2. Split into two migrations: add the field as nullable, "
-                f"write a data migration to populate existing rows, then alter "
-                f"to NOT NULL:\n"
+                f"  2. Add the field with allow_null=True, scaffold a data "
+                f"migration to populate existing rows, then remove allow_null=True "
+                f"— convergence applies NOT NULL on the next sync:\n"
                 f"       uv run plain migrations create --empty --name backfill_{field_name}"
             )
         self.add_operation(
@@ -898,6 +909,16 @@ def generate_altered_fields(self) -> None:
                     new_field.remote_field.through = old_field.remote_field.through  # ty: ignore[unresolved-attribute]
             old_field_dec = self.deep_deconstruct(old_field)
             new_field_dec = self.deep_deconstruct(new_field)
+            # Exclude non_migration_attrs (allow_null, default, on_delete, choices,
+            # etc.) — convergence handles those transitions, and the schema
+            # editor short-circuits when only non_migration_attrs differ. Emitting
+            # an AlterField here would be a no-op at apply time.
+            old_field_dec = _strip_non_migration_attrs(
+                old_field_dec, old_field.non_migration_attrs
+            )
+            new_field_dec = _strip_non_migration_attrs(
+                new_field_dec, new_field.non_migration_attrs
+            )
             if old_field_dec != new_field_dec and old_field_name == field_name:
                 both_m2m = isinstance(old_field, ManyToManyField) and isinstance(
                     new_field, ManyToManyField
@@ -907,26 +928,6 @@ def generate_altered_fields(self) -> None:
                 ) and not isinstance(new_field, ManyToManyField)
                 if both_m2m or neither_m2m:
                     # Either both fields are m2m or neither is
-                    if (
-                        isinstance(old_field, ColumnField)
-                        and isinstance(new_field, ColumnField)
-                        and old_field.allow_null
-                        and not new_field.allow_null
-                        and not new_field.has_persistent_column_default()
-                    ):
-                        raise MigrationSchemaError(
-                            f"Cannot alter field '{model_name}.{field_name}' "
-                            f"from nullable to NOT NULL without a default. "
-                            f"Existing NULL rows have no value to fall back "
-                            f"on.\n\n"
-                            f"Choose one:\n"
-                            f"  1. Declare a default on the field in models.py, e.g.\n"
-                            f'       {field_name}: str = types.TextField(default="...")\n'
-                            f"  2. Split into two migrations: write a data "
-                            f"migration to populate the NULL rows first, then "
-                            f"alter to NOT NULL:\n"
-                            f"       uv run plain migrations create --empty --name backfill_{field_name}"
-                        )
                     self.add_operation(
                         package_label,
                         operations.AlterField(