You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
PR #652 closed #566 for PostgreSQL only and explicitly deferred the analogous MySQL, MongoDB and BigQuery mappings. Their create handlers still parse only --table-mapping source:target and populate required identifiers, defaulting or omitting every advanced mapping field. #593 covers source settings/TLS; #594 covers the streaming destination table; neither covers these database-source mappings.
Scope and command design
Adopt the existing repeatable --table-mapping-json <JSON> on cloud clickpipe create mysql|mongodb|bigquery, combinable with the simple mapping flag using PostgreSQL's ordering and error semantics.
MySQL: ClickPipeMySQLPipeTableMapping: sourceSchemaName/sourceTable/targetTable plus excludedColumns, useCustomSortingKey, sortingKeys, tableEngine, partitionKey and partitionByExpr added by OpenAPI drift: 108 gaps between live spec and library #522. Explain that partitionKey controls snapshot partitioning while partitionByExpr defines destination PARTITION BY.
MongoDB: ClickPipeMongoDBPipeTableMapping: sourceDatabaseName/sourceCollection/targetTable and tableEngine.
BigQuery: ClickPipeBigQueryPipeTableMapping: sourceDatasetName/sourceTable/targetTable plus excludedColumns, useCustomSortingKey, sortingKeys and tableEngine. Preserve omission for the optional fields corrected by OpenAPI drift: 108 gaps between live spec and library #522.
Extend the PostgreSQL JSON-mapping approach with source-specific allowed wire fields and typed enum validation. Reuse mechanics while preserving each source's field names, requiredness and optionality; do not copy PostgreSQL's mandatory default values into the other types.
Validate before network/file reads: JSON object, known keys, nonempty required names, known engine, no empty list entries, documented uniqueness, and the sortingKeys/useCustomSortingKey relationship. Keep simple-form request behavior compatible except for optionality fixes intentionally landed in OpenAPI drift: 108 gaps between live spec and library #522.
Each source accepts simple-only, JSON-only and mixed input; preserves deterministic ordering; serializes minimal/maximal exact mapping objects; rejects malformed/unknown/missing fields and contradictory sorting flags with usage exit 2 and no request; exercises every supported engine; keeps absent BigQuery options off the wire. Cover MySQL partitionByExpr separately from partitionKey. README gives one meaningful example per source.
Implementation conventions: place clap definitions, request builders, CloudClient wrappers and dispatch in the owning CLI domain; map library errors through CloudClient with organization context. Classify writes exhaustively and fail fast for OAuth writes. Preserve --json/agent output, tolerate missing response fields, and use print_human for detail output. Add README examples. Update both CI classifiers for new files. Validate parsing and minimal/maximal builders, real-binary wiremock auth/request/error/output paths, cargo fmt, default and telemetry-disabled Clippy/check, and tests for every touched crate.
Problem and evidence
PR #652 closed #566 for PostgreSQL only and explicitly deferred the analogous MySQL, MongoDB and BigQuery mappings. Their create handlers still parse only
--table-mapping source:targetand populate required identifiers, defaulting or omitting every advanced mapping field. #593 covers source settings/TLS; #594 covers the streaming destination table; neither covers these database-source mappings.Scope and command design
Adopt the existing repeatable
--table-mapping-json <JSON>oncloud clickpipe create mysql|mongodb|bigquery, combinable with the simple mapping flag using PostgreSQL's ordering and error semantics.ClickPipeMySQLPipeTableMapping: sourceSchemaName/sourceTable/targetTable plus excludedColumns, useCustomSortingKey, sortingKeys, tableEngine, partitionKey and partitionByExpr added by OpenAPI drift: 108 gaps between live spec and library #522. Explain that partitionKey controls snapshot partitioning while partitionByExpr defines destination PARTITION BY.ClickPipeMongoDBPipeTableMapping: sourceDatabaseName/sourceCollection/targetTable and tableEngine.ClickPipeBigQueryPipeTableMapping: sourceDatasetName/sourceTable/targetTable plus excludedColumns, useCustomSortingKey, sortingKeys and tableEngine. Preserve omission for the optional fields corrected by OpenAPI drift: 108 gaps between live spec and library #522.Dependencies and implementation
clickpipe updatefor post-create source changes (Postgres/MySQL CDC settings, table mappings, connection) #569 table-add/remove operations without expanding PATCH beyond its schema.Acceptance and tests
Each source accepts simple-only, JSON-only and mixed input; preserves deterministic ordering; serializes minimal/maximal exact mapping objects; rejects malformed/unknown/missing fields and contradictory sorting flags with usage exit 2 and no request; exercises every supported engine; keeps absent BigQuery options off the wire. Cover MySQL partitionByExpr separately from partitionKey. README gives one meaningful example per source.
Implementation conventions: place clap definitions, request builders, CloudClient wrappers and dispatch in the owning CLI domain; map library errors through CloudClient with organization context. Classify writes exhaustively and fail fast for OAuth writes. Preserve --json/agent output, tolerate missing response fields, and use print_human for detail output. Add README examples. Update both CI classifiers for new files. Validate parsing and minimal/maximal builders, real-binary wiremock auth/request/error/output paths, cargo fmt, default and telemetry-disabled Clippy/check, and tests for every touched crate.