Skip to content

Expose table mapping options for MySQL, MongoDB and BigQuery ClickPipes #691

Description

@sdairs

Problem and evidence

PR #652 closed #566 for PostgreSQL only and explicitly deferred the analogous MySQL, MongoDB and BigQuery mappings. Their create handlers still parse only --table-mapping source:target and populate required identifiers, defaulting or omitting every advanced mapping field. #593 covers source settings/TLS; #594 covers the streaming destination table; neither covers these database-source mappings.

Scope and command design

Adopt the existing repeatable --table-mapping-json <JSON> on cloud clickpipe create mysql|mongodb|bigquery, combinable with the simple mapping flag using PostgreSQL's ordering and error semantics.

  • MySQL: ClickPipeMySQLPipeTableMapping: sourceSchemaName/sourceTable/targetTable plus excludedColumns, useCustomSortingKey, sortingKeys, tableEngine, partitionKey and partitionByExpr added by OpenAPI drift: 108 gaps between live spec and library #522. Explain that partitionKey controls snapshot partitioning while partitionByExpr defines destination PARTITION BY.
  • MongoDB: ClickPipeMongoDBPipeTableMapping: sourceDatabaseName/sourceCollection/targetTable and tableEngine.
  • BigQuery: ClickPipeBigQueryPipeTableMapping: sourceDatasetName/sourceTable/targetTable plus excludedColumns, useCustomSortingKey, sortingKeys and tableEngine. Preserve omission for the optional fields corrected by OpenAPI drift: 108 gaps between live spec and library #522.

Dependencies and implementation

  1. Merge OpenAPI drift: 108 gaps between live spec and library #522 for MySQL partitionByExpr and BigQuery optionality; coordinate with BigQuery ClickPipe create sends zero-valued settings on every request #592's BigQuery settings work.
  2. Extend the PostgreSQL JSON-mapping approach with source-specific allowed wire fields and typed enum validation. Reuse mechanics while preserving each source's field names, requiredness and optionality; do not copy PostgreSQL's mandatory default values into the other types.
  3. Validate before network/file reads: JSON object, known keys, nonempty required names, known engine, no empty list entries, documented uniqueness, and the sortingKeys/useCustomSortingKey relationship. Keep simple-form request behavior compatible except for optionality fixes intentionally landed in OpenAPI drift: 108 gaps between live spec and library #522.
  4. Extract MongoDB and BigQuery pure create builders if necessary. Make mapping helpers reusable by Add clickpipe update for post-create source changes (Postgres/MySQL CDC settings, table mappings, connection) #569 table-add/remove operations without expanding PATCH beyond its schema.

Acceptance and tests

Each source accepts simple-only, JSON-only and mixed input; preserves deterministic ordering; serializes minimal/maximal exact mapping objects; rejects malformed/unknown/missing fields and contradictory sorting flags with usage exit 2 and no request; exercises every supported engine; keeps absent BigQuery options off the wire. Cover MySQL partitionByExpr separately from partitionKey. README gives one meaningful example per source.

Implementation conventions: place clap definitions, request builders, CloudClient wrappers and dispatch in the owning CLI domain; map library errors through CloudClient with organization context. Classify writes exhaustively and fail fast for OAuth writes. Preserve --json/agent output, tolerate missing response fields, and use print_human for detail output. Add README examples. Update both CI classifiers for new files. Validate parsing and minimal/maximal builders, real-binary wiremock auth/request/error/output paths, cargo fmt, default and telemetry-disabled Clippy/check, and tests for every touched crate.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    cloudAffects cloud CLI commandsenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions