Propagate DDL commands to workers v2

[ozgune]

Citus 5.0 propagates Alter Table and Create Index commands to worker nodes. We implemented this feature using Citus' current replication model. We also decided to switch to using 2PC (or pg_paxos) once the metadata propagation changes were implemented.

This issue tracks v2 of the DDL propagation changes and depends on #19.

[ozgune]

We have 4 other issues that relate to propagating DDL changes. For now, I'm using this issue as a high level one and documenting other issues below.

• Issue Extend DDL commands such as Alter, Delete or Create Table to support table partitioning. Modify parsing stage to handle these modifications. #168 talks about extending DDL commands for Alter, Delete, and Create Table to support table sharding. I'm happy to at least remove the Alter Table part from this issue. @samay-sharma, what do you think?
• Issue Make shard creation atomic #265 relates to failure handling for DDL command propagation. We decided to think about this as part of v2 changes (Propagate DDL commands to workers v2 #25).
• Issue Indexes can't be created without specifying the index name explicitly for distributed tables #312 relates to propagating CREATE INDEX commands that don't have index names.
• Issue CREATE INDEX CONCURRENTLY is not supported for distributed tables. #1007 relates to running CREATE INDEX CONCURRENTLY.
• Issue Add support for Alter Table Add/Drop Constraint #1239 relates to adding support for ALTER TABLE ADD/DROP CONSTRAINT commands.

After release, we must remember to reply to citusdata/pg_shard#98 to direct future readers to this project so they can benefit from the DDL stuff it provides. At that point, we can close that ticket.

The following 4 issues don't directly relate to DDL command propagation, but are worth mentioning nonetheless.

• Issue In worker_apply_shard_ddl_command(), we pretend that we have support for Create Unlogged Table #54 and Properly generate table DDL strings for unlogged tables. We currently don't differentiate between regular and unlogged tables. #184 talks about DDL propagation for Unlogged Tables
• Issue Ensure in DDL that if a table has replication enabled, then it also defines a primary key. #192 talks about DDL commands and defining primary keys
• Issue Distributed materialized views #40 talks about creating distributed materialized views

The following issues were related to DDL propagation. They have since been resolved:

• Issue Send DDL commands in parallel to worker nodes. #131 improves DDL propagation's performance by running these commands in parallel.
• Issue Allow users to run unsupported DDL statements manually #274 touches on a potential usability issue with our current behavior -- for Citus pre-4.1 customers who built their own DDL propagation scripts.
• Issue Error out / warn on ALTER TABLE SET SCHEMA and ALTER TABLE ALL IN TABLESPACE #132 relates adding error or warning messages for unsupported ALTER TABLE commands.
• Issue Make sure Truncate on distributed tables work as expected #86 relates to propagating TRUNCATE to shards.

[samay-sharma]

Adding 2 more issues which we should evaluate during DDL propagation v2:

#356 relates to avoiding passing the text query string directly to the worker nodes.
#357 relates to considering if we should run the command on the master before the workers.

[ozgune]

Adding two more issues to evaluate during DDL propagation v2:

#354 relates to improving DDL propagation support for index related commands.
#376 relates to improving DDL propagation support for database related commands.

[lithp]

#513 also relates to this issue. ALTER TABLE ... SET NOT NULL can have partial failures if some shards have rows with null values

[onderkalaci]

I have a question here. Are there any blockers not to support CONCURRENTLY for index creation?

[aamederen]

#633 relates to improving DDL propagation support for rename related commands.

[ozgune]

@onderkalaci -- We're now tracking CREATE INDEX CONCURRENTLY in #1007.

(edited based on @anarazel's comment below)

The reason we currently don't support it is failure semantics. With CREATE INDEX, we can issue the command within a transaction block. If we observe a failure, we can roll it back. CREATE INDEX CONCURRENTLY doesn't run inside a transaction, so we may need to incorporate more logic to handle out-of-band errors.

[anarazel]

On 2016-12-20 12:00:03 -0800, Ozgun Erdogan wrote: @onderkalaci -- We're now tracking ``CREATE INDEX CONCURRENTLY`` in #1007. The reason we currently don't support it is failure semantics. With ``CREATE INDEX``, we can issue the command and wait for it to complete or error out. ``CREATE INDEX CONCURRENTLY`` runs in the background, so we may need to incorporate more logic to handle out-of-band errors.

CREATE INDEX CONCURRENTLY doesn't run in the background? Isn't the issue that you can't run it in a transaction, so we can't rollback if one of the CREATE INDEXes fails?

[marcocitus]

Most of the issues referenced here are closed. DDL is now transactional, parallel, and supports many more commands, including CREATE INDEX CONCURRENTLY. We can use individual issues to track missing features (or open a new parent issue).