Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: _cq_sync_time for long lived processes #15690

Open
1 task
bbernays opened this issue Dec 13, 2023 · 0 comments
Open
1 task

feat: _cq_sync_time for long lived processes #15690

bbernays opened this issue Dec 13, 2023 · 0 comments

Comments

@bbernays
Copy link
Collaborator

Which problem is this feature request solving?

Currently for long lived syncing processes like AWS event-based sync, the _cq_sync_time reflects the time that the process began not which is much less meaningful when syncs can last days or weeks.

Describe the solution you'd like

There are a few options:

  1. Enable the plugin to set the _cq_sync_time and have the CLI respect that value and not add another column/field
  2. Add a _cq_last_updated timestamp value that represents the last time the record was updated. For append mode this column would never be updated, but for update and delete-stale it would be updated every time the record is updated. The Value of the timestamp would either be set by the destination plugin at the time it was added to the batch

Pull request (optional)

  • I can submit a pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: No status
Development

No branches or pull requests

2 participants