feat: Add --ddl-file option and support offload to an existing empty table

[nj1973]

DDL file
This PR adds a new option --ddl-file that will write CREATE TABLE DDL to a file and exit the Offload command without copying data. The option supports writing the file to local storage or cloud storage. Based on comments on this PR the DDL file also includes DDL to support the --create-backend-db option.

The basic use case is output the DDL for a table that has not yet been offloaded at all. We've plugged it into the existing flow at the part where table DDL is generated, therefore and SQL commands to create backend databases (--create-backend-database) or drop existing tables (--reset-backend-table) will still be visible in the output. Any command that runs in execute mode will be downgraded to non-execute mode.

You'll notice in the file changes that there are a lot of changes related to execute and dry_run. That is because I needed to move the execute flag from OrchestrationConfig to OffloadOperation. I'd tried to do this previously but was thwarted by Incremental Update. Now that Incremental Update has been removed I was able to make this change. I needed the change because --ddl-file will switch a command from execute=True to execute=False and changing it in config was the wrong thing to do.

Offload to existing table
This PR also adds support for Offloading to a pre-created table - but only if the table is empty and has no Offload metadata.

If the table has data then the normal --reset-backend-table option is required.

If the table is empty but has metadata (for example a sysadmin may have truncated the backend table) then the new --reuse-backend-table option is required.

[abb9979]

Are the trailing spaces in a DDL statement necessary?

[abb9979]

One of my tests was to re-use an existing DDL file.

Option error: 'DDL path already exists: /tmp/t3.sql'

Makes me wonder whether there should be a --reuse-ddl-file option?

[nj1973]

Are the trailing spaces in a DDL statement necessary?

Not necessary no, I've not changed the SQL text generation, only redirected it to a file. I suppose we should look to address this though because I wouldn't like trailing spaces in code in my repo.

[nj1973]

One of my tests was to re-use an existing DDL file.

Option error: 'DDL path already exists: /tmp/t3.sql'

Makes me wonder whether there should be a --reuse-ddl-file option?

I don't think so, I think that is one option too many. We already have too many. We should either abort as above or overwrite the file. I chose abort because it is safer.

[abb9979]

Data type modifications are not reflected in the DDL file. For example --variable-string-columns=created ignored and DDL file still has this column as DATETIME.

[abb9979]

Using the --reuse-backend-table and --reset-backend-table options together gives the latter precedence and drops the backend table. I think this combination should be a message to the user to choose the appropriate option only.

[abb9979]

Screen output is minimal and shows nothing for short-circuited (i.e. prevented) offloads. Includes, for example, trying to --reuse-backend-table when it is populated.

[abb9979]

If a pre-existing backend table has a different column type to the frontend table, the mismatch is raised, even if an offload override option is used to bring them into alignment.

[nj1973]

Are the trailing spaces in a DDL statement necessary?

These have been removed.

[nj1973]

Using the --reuse-backend-table and --reset-backend-table options together gives the latter precedence and drops the backend table. I think this combination should be a message to the user to choose the appropriate option only.

Fixed, thanks

[nj1973]

Screen output is minimal and shows nothing for short-circuited (i.e. prevented) offloads. Includes, for example, trying to --reuse-backend-table when it is populated.

Fixed, thanks