Postgres source: check if Debezium is active

[tuliren]

Tell us about the problem you're trying to solve

• Currently we wait for Debezium to start processing the relevant logs without a timeout. The timeout is 5 minutes.
• This approach does not work in two ways.
  1. Sometimes Debezium has too many processed logs to go through. So 5 minutes is not enough.
  2. The connector will always wait for Debezium for 5 minutes even when there is no data to sync. This is unnecessary overhead for data syncing. Also it slows down the integration test significantly. For now, the whole Postgres source integration test take 40minutes to one hour.

Describe the solution you’d like

• From the logs, we know that Debezium usually starts very quickly. From my local observation, it only takes 17 seconds. When Debezium is active, it actively emits logs informing us what it is doing.
• If we can somehow get the information from Debezium, we don't need a hardcoded timeout. We can simply check whether Debezium is still active or not.
  • The check can either be listening to changes in the state file, or calling a Debezium API about its latest status, if such an API exists.
• If it is possible, reduce the timeout from 5 minutes to 1 minute.

Describe the alternative you’ve considered or used

N/A

Additional context

N/A

[tuliren]

Connect to MySQL RDS CDC can be 1-2 minutes. The 5 minutes timeout is inspired by MySQL, not Postgres.

[grishick]

Next steps:

• Move this back into backlog
• file an issue to log LSN diff
• file an issue to wait while Debezium is making a snapshot

Other options to consider: add this feature to Debezium

[grishick]

Going to postpone this for now, because there is no non-hacky implementation path for this at this point.

[tuliren]

Created a Debezium issue here: https://issues.redhat.com/browse/DBZ-5432

[bleonard]

Hopeful for this approach #15040

[rodireich]

This should be resolved by #15040 / #18987