table.transform() method for advanced alter table

[simonw]

SQLite's ALTER TABLE can only do the following:

• Rename a table
• Rename a column
• Add a column

Notably, it cannot drop columns - so tricks like "add a float version of this text column, populate it, then drop the old one and rename" won't work.

The docs here https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes describe a way of implementing full alters safely within a transaction, but it's fiddly.

1. Create new table
2. Copy data
3. Drop old table
4. Rename new into old

It would be great if sqlite-utils provided an abstraction to help make these kinds of changes safely.

[simonw]

Idea: use a chained API to define a complex transition and then execute it all at once. For example:

db["mytable"].transform().rename("col1", "col_1") \
  .change_type("col1", float) \
  .execute()

[simonw]

Ideally this would all happen in a single transaction, such that other processes talking to the database would not see any inconsistent state while the table copy was taking place. Need to confirm that this is possible. Also refs transactions thoughts in #121.

[simonw]

I'm not so keen on that chained API - it's pretty complicated.

Here's an idea for a much simpler interface. Essentially it lets you say "take table X and migrate its contents to a new table with this structure - then atomically rename the tables to switch them":

db["mytable"].migrate_table({"id": int, "name": str"}, pk="id")

The migrate_table() method would take the same exact signature as the table.create() method:

sqlite-utils/sqlite_utils/db.py

Lines 615 to 625 in a236a6b

	def create(
	self,
	columns,
	pk=None,
	foreign_keys=None,
	column_order=None,
	not_null=None,
	defaults=None,
	hash_id=None,
	extracts=None,
	):

[simonw]

Alternative possible names:

• .transform_table()
• .migrate()
• .transform()

I'm torn between .migrate_table() and .transform_table().

[simonw]

Since neither the term "transform" or "migrate" are used in the codebase at the moment, I think I'll go with .transform_table() - that leaves the term "migrate" available for any future database migrations system (similar to Django's).

[simonw]

Don't forget this step:

If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF.

[simonw]

Thinking about the method signature:

    def transform_table(
        self,
        columns,
        pk=None,
        foreign_keys=None,
        column_order=None,
        not_null=None,
        defaults=None,
        hash_id=None,
        extracts=None,
    ):

This requires the caller to provide the exact set of columns for the new table.

It would be useful if this was optional - if you could omit the columns and have it automatically use the previous columns. This would let you change things like the primary key or the column order using the other arguments.

Even better: allow column renaming using an optional rename={...} argument:

db["dogs"].transform_table(rename={"name": "dog_name"})

[simonw]

I can have a convenient change_type={...} parameter for changing column types too.

[simonw]

Work in progress: not quite right yet, I need smarter logic for how renamed columns are reflected in the generated INSERT INTO ... SELECT ... query:

    def transform_table(
        self,
        columns=None,
        rename=None,
        change_type=None,
        pk=None,
        foreign_keys=None,
        column_order=None,
        not_null=None,
        defaults=None,
        hash_id=None,
        extracts=None,
    ):
        assert self.exists(), "Cannot transform a table that doesn't exist yet"
        columns = columns or self.columns_dict
        if rename is not None or change_type is not None:
            columns = {rename.get(key, key): change_type.get(key, value) for key, value in columns.items()}
        new_table_name = "{}_new_{}".format(self.name, os.urandom(6).hex())
        previous_columns = set(self.columns_dict.keys())
        with self.db.conn:
            columns = {name: value for (name, value) in columns.items()}
            new_table = self.db.create_table(
                new_table_name,
                columns,
                pk=pk,
                foreign_keys=foreign_keys,
                column_order=column_order,
                not_null=not_null,
                defaults=defaults,
                hash_id=hash_id,
                extracts=extracts,
            )
            # Copy across data - but only for columns that exist in both
            new_columns = set(columns.keys())
            columns_to_copy = new_columns.intersection(previous_columns)
            copy_sql = "INSERT INTO [{new_table}] ({new_cols}) SELECT {old_cols} FROM [{old_table}]".format(
                new_table=new_table_name,
                old_table=self.name,
                old_cols=", ".join("[{}]".format(col) for col in columns_to_copy),
                new_cols=", ".join("[{}]".format(rename.get(col, col)) for col in columns_to_copy),
            )
            self.db.conn.execute(copy_sql)
            # Drop the old table
            self.db.conn.execute("DROP TABLE [{}]".format(self.name))
            # Rename the new one
            self.db.conn.execute(
                "ALTER TABLE [{}] RENAME TO [{}]".format(new_table_name, self.name)
            )
        return self

[simonw]

According to https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes the hardest bits to consider are how to deal with existing foreign key relationships, triggers and views.

I'm OK leaving views as an exercise for the caller - many of these transformations may not need any view changes at all.

Foreign key relationships are important: it should handle these automatically as effectively as possible.

Likewise trigger changes: need to think about what this means.

[simonw]

I'm going to add a second method .transform_table_sql(...) - which returns the SQL that would have been executed but does NOT execute it.

Advanced callers can use this to include their own additional steps in the same transaction - e.g. recreating views or triggers.

More importantly it gives me a useful hook for writing some unit tests against the generated SQL.

[simonw]

Work in progress in transform branch here: https://github.com/simonw/sqlite-utils/tree/transform