New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🐛 Source Insightly: fix remove timedelta for incremental causing missing records #24422
🐛 Source Insightly: fix remove timedelta for incremental causing missing records #24422
Conversation
Signed-off-by: Henri Blancke <blanckehenri@gmail.com>
Signed-off-by: Henri Blancke <blanckehenri@gmail.com>
Signed-off-by: Henri Blancke <blanckehenri@gmail.com>
/test connector=connectors/source-insightly
Build PassedTest summary info:
|
Thanks for the PR @henriblancke this looks good to me. For future reference, in case someone does want to try to fix the connector so that it doesn't produce duplicate records (while also not producing missing records), what can be done to figure out whether or not the |
/publish connector=connectors/source-insightly
if you have connectors that successfully published but failed definition generation, follow step 4 here |
@henriblancke can you fix the merge conflicts here? I'm guessing there needs to be a rebase with master. |
Signed-off-by: Henri Blancke <blanckehenri@gmail.com>
@sajarin I'll have our team reach out to their tech team and hopefully get a clear answer on the expected behavior.
Should be rebased and merge conflicts are fixed |
Thanks for the contribution @henriblancke! |
What
Adding 1 second to the cursor value can occasionally cause missing records. There is no clear indication in the insightly docs if providing an
updated_after_utc
is>
or>=
. In any case having an occasional duplicate is better than missing records.How
Don't add 1 second to the cursor value before retrieving data incrementally
Recommended reading order
x.java
y.python
🚨 User Impact 🚨
Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.
Pre-merge Checklist
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.0.0.1
Dockerfile
has version0.0.1
README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog with an entry for the initial version. See changelog exampledocs/integrations/README.md
airbyte-integrations/builds.md
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereUpdating a connector
Community member or Airbyter
Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with
airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.Code reviews completed
Connector version has been incremented
Dockerfile
has updated versionDocumentation updated
README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
with an entry for the new version. See changelog examplePR name follows PR naming conventions
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereConnector Generator
-scaffold
in their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates
then checking in your changes