Internal schema change CQL queries should not be used for distributed tables

[psarna]

As @nyh suggested in #6513, I'm creating a separate issue for a thing that came up in the discussion.

Internal CQL queries are generally expected to be used for local tables, not regular, distributed ones. One of the reasons is that any schema changes issued by internal CQL queries are only propagated locally, which means that calling ALTER TABLE on a regular, distributed table will result in not propagating this change to other nodes, which leads to schema inconsistencies. There should be an assertion/warning somewhere that fires when somebody tries to call an internal query on a regular, distributed table. Such misuse (of mine) was the root cause of #6513.

Example internal query:

    sstring req = format("ALTER TABLE \"{}\".\"{}\" WITH {} = {}",
            schema->ks_name(), schema->cf_name(), tags_extension::NAME, serialized_tags_str);
    return db::execute_cql(std::move(req)).discard_result();

[psarna]

Let's reopen - altering a schema is already guarded, but there are more statements that probably shouldn't be called from within the database - like CREATE TABLE, CREATE MATERIALIZED VIEW, etc.

[slivne]

@psarna are you working on a fix ?

@tgrabiec any comments on this ?

[psarna]

I wasn't actively working on it, but it's a simple thing, so I'll squeeze it in.

[avikivity]

Not a bug, not backporting.