Skip to content

Experimental: End sync as soon as current VGTID is equal to, or after desired stop VGTID

Pre-release
Pre-release

Choose a tag to compare

@notfelineit notfelineit released this 29 Jan 03:07
· 94 commits to main since this release
907e108

Main change

This release is intended to be upgraded to "latest release" after verifying in production that it works as intended:

  • If we reach a current VGTID that is equal to, or after, the desired stop VGTID, end the sync and flush records to Airbyte

How this is different

This differs from the implementations prior to 1.35.0, where the sync will look for a current VGTID that is exactly the desired stop VGTID.

However, sometimes because writes to the database continue to increase the current VGTID, by the time we request records from VStream, the current VGTID has already advanced past the desired stop VGTID. Therefore, the loop does not end.

Instead, the connector retries again with a new stop VGTID, and a new sync request. The new stop VGTID and new current VGTID will only be equal at some point where writes have slowed down, and no writes occur between the time of 1) fetching stop VGTID and 2) fetching current VGTID.

PRs

Full Changelog: v1.34.0...v1.35.0-experiemental