Decide CLI exposure for new API fields from OpenAPI drift #287

[sdairs]

#287 (resolved by #288) added 21 fields to clickhouse-cloud-api at the library level only. Per the drift workflow, deciding which of them to surface in the CLI is a separate call. Candidates:

• chctl cloud clickpipe create (object storage): --skip-initial-load (skip initial load, ingest only queue-notification files; only valid with --queue-url) and --start-after <object-key> (start continuous ingestion after this key; mutually exclusive with --skip-initial-load). Both also exist on the PATCH source for clickpipe update.
• chctl cloud service create: --min-replicas, --max-replicas, --replica-memory-gb for horizontal autoscaling (fixed memory per replica, variable replica count). Mutually exclusive with the existing --num-replicas/--min-replica-memory-gb/--max-replica-memory-gb flags, and requires horizontal autoscaling to be enabled for the organization — flag validation should enforce the exclusivity client-side.
• Scaling schedule entries: numReplicas (fixed count for vertical windows) and replicaMemoryGb (fixed memory for horizontal windows) on ScalingScheduleEntryRequest. The CLI currently has no surface that constructs schedule entries, so this only matters if/when one is added.

Per CLAUDE.md, the CLI does not need 100% endpoint/field coverage — be intentional about what is exposed.

🤖 Generated with Claude Code

[sdairs]

Updated plan: CLI exposure for #299 + remaining #288 fields

Researching the current CLI surface (at origin/main, which includes #299 and #300) against the library. The original plan in the previous comment covered only #288's 21 fields. Since then:

• Fix OpenAPI drift: 36 gaps between live spec and library (#298) #299 ("Fix OpenAPI drift: 36 gaps") added a new Beta endpoint (click_pipe_schema_discovery), 4 new models, an AutoscalingMode enum, serverId on MySQL sources, autoscalingMode on 7 structs, and 9 enum values — while removing replicaMemoryGb from ServicePostRequest / ServiceReplicaScalingPatchRequest / ScalingScheduleEntry.
• Derive CLI value lists from library enums with analyzer drift guard #300 ("Derive CLI value lists from library enums") shipped VALUES consts on the input enums and wired the CLI to validate against them, which already reconciles the ca-central-1 region additions and the europe-west2 gap it called out. No CLI work remains for region/enum-value drift.

This supersedes the autoscaling section of the previous comment: replicaMemoryGb is no longer on the request structs (confirmed removed by #299), and autoscalingMode is now a typed library enum that should drive the new --autoscaling-mode flag.

Already reconciled — no CLI work

• ca-central-1 on 8 region enums (Fix OpenAPI drift: 36 gaps between live spec and library (#298) #299): ServicePostRequestRegion is validated in the CLI via ServicePostRequestRegion::VALUES (Derive CLI value lists from library enums with analyzer drift guard #300), which includes ca-central-1. The other 7 region enums (ServiceRegion, ServiceScalingPatchResponseRegion, ByocConfigRegionid, ByocInfrastructurePostRequestRegionid, InstancePrivateEndpointRegion, OrganizationPrivateEndpointRegion, OrganizationPatchPrivateEndpointRegion) are response-only or parsed through serde_json::from_value with an Unknown catch-all, so they accept new wire values without CLI changes.
• stopped on PgStateProperty (Fix OpenAPI drift: 36 gaps between live spec and library (#298) #299): the Postgres state label is rendered via serde_json::to_value, and the enum has the Unknown catch-all. Stopped is now a named variant. Already handled.
• europe-west2 (Derive CLI value lists from library enums with analyzer drift guard #300): already added to ServicePostRequestRegion::VALUES; CLI accepts it.
• PgConfig autovacuum_* (Resolve OpenAPI drift: object storage ingestion controls, pg autovacuum config, horizontal autoscaling fields (#287) #288): already flows through the generic postgres config patch --set autovacuum_max_workers=3 mechanism. No CLI work.
• Scaling schedule entries (ScalingScheduleEntry / ScalingScheduleEntryRequest): no service scaling-schedule command exists; constructing schedule entries is a new feature surface, out of scope for "decide CLI exposure of drift fields." Leave library-only. (Note: Fix OpenAPI drift: 36 gaps between live spec and library (#298) #299 removed numReplicas/replicaMemoryGb from ScalingScheduleEntry and added autoscalingMode — still library-only either way.)

Needs CLI exposure

1. ClickPipe object-storage create — --skip-initial-load / --start-after (from #288, unchanged)

ObjectStorageCreateArgs (src/cloud/cli.rs) already exposes --queue-url but not the two ingestion-control fields that gate on it. Library structs ClickPipeObjectStorageSource and ClickPipePatchObjectStorageSource both carry skipInitialLoad / startAfter.

• Add --skip-initial-load (flag, bool) with requires = "queue_url" (spec: "Only applicable when queueUrl is provided").
• Add --start-after <object-key> (Option<String>) with conflicts_with = "skip_initial_load" (spec: "Cannot be provided when skipInitialLoad is true").
• Wire in commands.rs::clickpipe_create_object_storage (replace the two None placeholders).
• No main.rs change (whole struct forwarded).

2. Service create + scale horizontal autoscaling — --min-replicas / --max-replicas / --autoscaling-mode (revised for #299)

ServicePostRequest and ServiceReplicaScalingPatchRequest now carry autoscalingMode: Option<AutoscalingMode> (typed enum, vertical default) plus minReplicas / maxReplicas (the horizontal pair). The vertical pair (numReplicas / minReplicaMemoryGb / maxReplicaMemoryGb) already exists in the CLI. build_create_service_request and service_scale currently hardcode autoscaling_mode: None, min_replicas: None, max_replicas: None.

service create (ServiceCommands::Create, cli.rs) and service scale (ServiceCommands::Scale, cli.rs):

• Add --min-replicas, --max-replicas (both Option<u32>).
• Add --autoscaling-mode <vertical|horizontal> with value_parser = PossibleValuesParser::new(AutoscalingMode::VALUES). Decision: add the VALUES const to AutoscalingMode (pub const VALUES: &'static [&'static str] = &["vertical", "horizontal"]; in an impl AutoscalingMode block in models.rs), mirroring Derive CLI value lists from library enums with analyzer drift guard #300's pattern so the analyzer's EnumValuesMismatch guard keeps it spec-accurate going forward. No CLI-local literal.
• conflicts_with_all = ["num_replicas", "min_replica_memory_gb", "max_replica_memory_gb"] on each horizontal flag; symmetric conflicts_with_all on the vertical flags so vertical-only usage stays valid and mixing is a clap error.

commands.rs:

• CreateServiceOptions / ServiceScaleOptions: add min_replicas, max_replicas, autoscaling_mode.
• build_create_service_request: set min_replicas/max_replicas from opts; set autoscaling_mode (explicit flag wins; infer Horizontal when the horizontal pair is present and no explicit mode; None/Vertical otherwise). Replaces the three None placeholders (~line 659-662).
• service_scale: set min_replicas/max_replicas/autoscaling_mode on ServiceReplicaScalingPatchRequest (replacing ~line 1918-1919).
• Client-side validation: reject min-without-max (and vice versa) on create/scale; reject mixing horizontal + vertical (clap conflicts_with_all covers most, but min-without-max needs a builder check with a clear error). Org-feature enablement stays server-side — don't replicate.
• main.rs: update the two match arms (ServiceCommands::Create, ServiceCommands::Scale) to forward the new fields.

service get already renders horizontal fields via print_human (renders all non-empty serde fields), so no output changes needed.

3. ClickPipe MySQL serverId — --server-id (new in #299)

ClickPipeMutateMySQLSource (create), ClickPipeMySQLSource, and ClickPipePatchMySQLSource (update) now carry serverId: Option<i64> (1–4294967295, "Optional MySQL server_id the pipe declares itself as in the MySQL replication topology. Must be unique across replicas connected to the source. If omitted, one is assigned automatically."). The create handler currently hardcodes server_id: None.

• Add --server-id <N> (Option<u64>, clap value_parser clamped 1..=4294967295) to MySqlCreateArgs.
• Wire in commands.rs::clickpipe_create_mysql (replace server_id: None).
• No main.rs change (whole struct forwarded).

There is no clickpipe update source command today (only clickpipe settings update, which tunes ingestion settings, not the source). So ClickPipePatchMySQLSource.serverId has no CLI surface to attach to until a source-update command is added — out of scope here. Record it as a known gap.

4. ClickPipe schema discovery — new clickpipe schema-discover command (new in #299, Beta) — EXPOSE

Decision: expose now, with a (beta) label in the command about, mirroring the existing Postgres convention (/// Manage ClickHouse Cloud Postgres services (beta) at cli.rs:266). The CLI already marks an entire command group as beta this way; schema-discover is a single subcommand, so the (beta) note goes on the subcommand about.

#299 added click_pipe_schema_discovery (POST /v1/organizations/{orgId}/services/{serviceId}/clickpipes/schemaDiscovery, Beta) plus ClickPipeSchemaDiscoveryRequest / ClickPipeSchemaDiscoverySource / ClickPipeSchemaDiscoveryResponse / ClickPipeSchemaDiscoveryField. The request body takes a source with optional kafka (ClickPipePostKafkaSource) or kinesis (ClickPipePostKinesisSource) and returns inferred fields: [{name, type, optional}]. The Client method exists; nothing in the CLI calls it.

Reasons to expose:

• It's a read-only operation (infers a schema without creating a pipe), so it's safe under OAuth/Bearer auth.
• It directly helps users fill in --column "name:Type" on clickpipe create kafka|kinesis, which today they have to guess.
• The create handlers already build ClickPipePostKafkaSource / ClickPipePostKinesisSource, so the source-construction logic is reusable.

Cautions handled:

• Beta: label (beta) in the subcommand about (Postgres-group precedent). The library already tracks BETA_OPERATIONS; no new beta-surface policy needed for a single labeled subcommand.
• Source coverage: only Kafka and Kinesis are supported — not object storage / Postgres / MySQL / MongoDB / BigQuery. Reject unsupported source types up front (clap value_parser on --source) rather than letting the API return 400.

Proposed surface:

chctl cloud clickpipe schema-discover <service-id> --source kafka [kafka args...] [--json]
chctl cloud clickpipe schema-discover <service-id> --source kinesis [kinesis args...] [--json]

Implementation:

• New ClickPipeCommands::SchemaDiscover variant (or a SchemaDiscover subcommand) in cli.rs. Reuse KafkaCreateArgs / KinesisCreateArgs fields for source construction (or extract a build_kafka_source / build_kinesis_source helper from the create handlers so both call sites share it — currently the source is built inline in clickpipe_create_kafka/_kinesis).
• New CloudClient wrapper click_pipe_schema_discovery in client.rs (thin delegate to self.api().click_pipe_schema_discovery()).
• Handler in commands.rs: build a ClickPipeSchemaDiscoveryRequest with the chosen source, call the wrapper, print fields as a table (name, type, optional) or JSON.
• Classify SchemaDiscover in is_write_command() as read-only (it's a POST but side-effect-free; add to the read-only test list).
• --json output via the standard pattern.

Keep as group 4 in the implementation order below. It introduces a new command rather than just new flags, so it may land as its own PR for reviewability — but it is in scope for this plan, not deferred to a separate issue.

Tests

• cli.rs try_parse_from:
  • --skip-initial-load requires --queue-url; --start-after conflicts with --skip-initial-load; both parse together with --queue-url.
  • service create horizontal pair parses; horizontal + vertical fails (clap conflicts_with_all); --autoscaling-mode badvalue fails.
  • clickpipe create mysql --server-id 4242 parses; --server-id 0 and --server-id 4294967296 fail.
  • clickpipe schema-discover --source kafka parses; --source object-storage fails (if unsupported types are rejected at parse time).
• commands.rs build_create_service_request (minimal + maximal): minReplicas/maxReplicas/autoscalingMode: "horizontal" land on wire JSON when the horizontal pair is given; autoscalingMode absent and vertical fields set otherwise; mixing rejected with a clear error. service_scale analog.
• commands.rs clickpipe_create_mysql: serverId lands on wire JSON when --server-id given; absent otherwise.
• cli_request_shape_test.rs: assert POST wire JSON carries skipInitialLoad/startAfter when the clickpipe object-storage flags are passed; assert serverId on the MySQL create body.
• is_write_command: add SchemaDiscover to the read-only test list (if exposed).
• No new destructive test cases needed (create/scale already classified).

Docs

• README.md clickpipe object-storage create example: show --skip-initial-load / --start-after with the --queue-url continuous example.
• README.md service create table: add --min-replicas / --max-replicas / --autoscaling-mode rows noting horizontal autoscaling + org-feature requirement + mutual exclusivity with the vertical flags.
• README.md clickpipe mysql create: note --server-id.
• If schema discovery is exposed: README.md clickpipe section + a --help (beta) note.

Verification

• cargo test -p clickhouse-cloud-api -p clickhousectl
• cargo clippy -p clickhousectl --all-targets -- -D warnings
• cargo check --workspace --all-features
• python3 scripts/check-openapi-drift.py --dry-run (library unchanged → still no drift)
• Manual --help smoke check on the changed commands.

Implementation order

1. ClickPipe object-storage flags (group 1 — cleanest, library already has the fields).
2. Service horizontal-autoscaling flags (group 2 — needs the AutoscalingMode::VALUES const + validation).
3. ClickPipe MySQL --server-id (group 3 — trivial, single field).
4. ClickPipe schema discovery (group 4 — new command, Beta; label (beta) in the subcommand about; may land as its own PR for reviewability).

Decisions (confirmed)

• AutoscalingMode::VALUES: add the pub const VALUES to AutoscalingMode in models.rs, consistent with Derive CLI value lists from library enums with analyzer drift guard #300 and guarded by the analyzer's EnumValuesMismatch check. No CLI-local literal.
• Schema discovery exposure: expose now with a (beta) note in the subcommand about, following the existing Postgres-group convention (/// Manage ClickHouse Cloud Postgres services (beta) at cli.rs:266). No new beta-surface policy is required for a single labeled subcommand.