Skip to content

[INLONG-6896][Sort] PostgreSQL-CDC supports table level metrics#6897

Merged
gong merged 3 commits intoapache:masterfrom
liaorui:feature-postgres
Dec 16, 2022
Merged

[INLONG-6896][Sort] PostgreSQL-CDC supports table level metrics#6897
gong merged 3 commits intoapache:masterfrom
liaorui:feature-postgres

Conversation

@liaorui
Copy link
Contributor

@liaorui liaorui commented Dec 15, 2022

Prepare a Pull Request

Fixes #6896

Motivation

All tables in one database transferring scenario, postgres-cdc can upload metrics group by schema and table.

Modifications

  • DebeziumSourceFunction replaces SourceMetricData with SourceTableMetricData. Uses outputMetricsWithEstimate method to upload metrics group by schema and table.
  • PostgresSQLSource has a new migrateAll parameter. DebeziumSourceFunction will use migrateAll flag to determine if registering sub source metric data from metric state and uploading metrics group by schema and table.

Verifying this change

  • This change added tests and can be verified as follows:

    (example:)

    • Added integration tests for end-to-end deployment with large payloads (10MB)
    • Extended integration test for recovery after broker failure

Documentation

  • Does this pull request introduce a new feature? (no)

@gong gong changed the title [INLONG-6896][Sort]Postgres-CDC supports metrics by schema and table [INLONG-6896][Sort] Postgres-CDC supports table level metrics Dec 15, 2022
@healchow healchow changed the title [INLONG-6896][Sort] Postgres-CDC supports table level metrics [INLONG-6896][Sort] PostgreSQL-CDC supports table level metrics Dec 15, 2022
@gong gong merged commit 21b612a into apache:master Dec 16, 2022
featzhang pushed a commit to featzhang/inlong that referenced this pull request Dec 17, 2022
@liaorui liaorui deleted the feature-postgres branch December 23, 2022 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature][Sort] PostgreSQL-CDC supports metrics by database and table

4 participants