Support MySQL 8 high resolution replication timestamps from GTID events

[methodmissing]

References and depends on osheroff/mysql-binlog-connector-java#130

Gist of the changes

More accurate metrics and also source.ts_ms for MySQL >= 8.0.1 with GTID replication mode enabled. At Shopify we're doubling down on lower latency business use cases for CDC and we're aware of the second resolution (int4) timestamp constraint in binlog event headers for some time.

This issue and Github pull requests to the binlog client and Debezium is an attempt to engage with the community to report higher resolution metrics and source.ts_ms in a backwards compatible way.

Some example graphs from a lower environment connecting to MySQL 8 writers where the higher resolution timestamps are rendered - a lot easier to spot trends and variance for the writer -> binlog client hop.

Dead code removals

I could not find any references to these methods anymore:

• SourceInfo::setBinlogTimestampSeconds
• SourceInfo::getBinlogTimestampSeconds

Alignment of event time extraction in onEvent and handleEvent

I preferred to handle event timestamp extraction in MySqlStreamingChangeEventSource::onEvent as both handlers are called in order by the binlog client event listeners feature. That way emitted metrics values and source.ts_ms key off the same Instant instance.

Event timestamp logic

MySQL 8.0.1 introduced 2 new microsecond resolution timestamps on GTID events to track replication lag more accurately:

• immediate_commit_timestamp, microsecond resolution timestamp from the immediate master (or equal to original_commit_timestamp if connecting to master) since MySQL 8.0.1
• original_commit_timestamp, microsecond resolution timestamp of the commit on the originating master (or equal to original_commit_timestamp if connecting to master) since MySQL 8.0.1

original_commit_timestamp is representative of the original commit time on the MySQL writer.

The order of precedence is:

• IF GTID mode is enabled and GtidEventData::getOriginalCommitTimestamp returns a non-zero value (MySQL >= 8.0.1), use the high resolution timestamp as set by the writer
• Otherwise fallback to the second resolution timestamps

Other due diligence checks

• Snapshots set the current time in milliseconds for source.ts_ms already, I did not find any timestamp resolution issues with aligning on millisecond resolution for the streaming path as well

Jira issue: https://issues.redhat.com/projects/DBZ/issues/DBZ-7183

[github-actions]

Welcome as a new contributor to Debezium, @methodmissing. Reviewers, please add missing author name(s) and alias name(s) to the COPYRIGHT.txt and Aliases.txt respectively.

[github-actions]

Hi @methodmissing, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key.

[github-actions]

Hi @methodmissing, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key.

[github-actions]

Hi @methodmissing, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key.

[Naros]

Overall LGTM @methodmissing, just a few comments inine.

[Naros]

Hi @methodmissing, would you please rebase against the latest main? Thanks.

[methodmissing]

Rebased, expected to fail on missing symbol gtidEvent::getOriginalCommitTimestamp:

[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
Error:  COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
Error:  /home/runner/work/debezium/debezium/debezium-connector-mysql/src/main/java/io/debezium/connector/mysql/MySqlStreamingChangeEventSource.java:[230,51] cannot find symbol
  symbol:   method getOriginalCommitTimestamp()
  location: variable gtidEvent of type com.github.shyiko.mysql.binlog.event.GtidEventData

Requires 0.28.4 of the binlog client released here and version bump here. Unless there's a different and preferred path without releasing 0.28.4 first.

[Naros]

Thanks @methodmissing, that makes perfect sense. I'll look at the binlog PR and build locally. Appreciate the quick turn around.

[Naros]

Waits for the upstream MySQL binlog client 0.28.4.

Tested this locally with the upstream PR in the client, both MariaDB and MySQL passed with mysql-ci-gtids profile, so we'll run the full suite once the upstream client has been published. For now, I expect the tests to fail due to the missing artifact.

[Naros]

Applied, thanks @methodmissing !