Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🎉 Support incremental sync in Sendgrid #3512

Merged
merged 9 commits into from
May 25, 2021
Merged

🎉 Support incremental sync in Sendgrid #3512

merged 9 commits into from
May 25, 2021

Conversation

htrueman
Copy link
Contributor

@htrueman htrueman commented May 20, 2021 •
edited by sync-by-unito bot

What

Added incremental streams support to global_suppressions, blocks, bounces, invalid_emails and spam_reports streams.

How

  • Added incremental configured catalogs json configs
  • Updated source to support start_time config
  • Added SendgridStreamIncrementalMixin to be inherited by streams that support incremental sync

Pre-merge Checklist

  • Run integration tests
  • Publish Docker images

Recommended reading order

  1. Configured catalogs json files
  2. source.py
  3. streams.py

┆Issue is synchronized with this Asana task by Unito

@htrueman
Add incremental streams support.
d666988 
Updated config catalogs.
@htrueman htrueman self-assigned this May 20, 2021
@htrueman htrueman linked an issue May 20, 2021 that may be closed by this pull request
Support incremental sync in Sendgrid #1382
Closed
keu
keu approved these changes May 20, 2021
View reviewed changes
Copy link
Contributor

@keu keu left a comment •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a small comment, otherwise approved, did you enable acceptance tests for incremental?

@keu keu removed the request for review from arhip11 May 20, 2021 20:10
@htrueman
Copy link
Contributor Author

htrueman commented May 21, 2021 •
edited by github-actions bot

/test connector=source-sendgrid

🕑 source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/863995756
❌ source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/863995756

@htrueman
Copy link
Contributor Author

LGTM, just a small comment, otherwise approved, did you enable acceptance tests for incremental?

It's added for now

@htrueman htrueman force-pushed the htrueman/source-sendgrid-support-incremental-sync branch from 3c29eb6 to 574074a Compare May 21, 2021 14:31
@htrueman
Copy link
Contributor Author

htrueman commented May 21, 2021 •
edited by github-actions bot

/test connector=source-sendgrid

🕑 source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/864185641
❌ source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/864185641

@htrueman
Copy link
Contributor Author

htrueman commented May 21, 2021 •
edited by github-actions bot

/test connector=source-sendgrid

🕑 source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/864235914
❌ source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/864235914

@htrueman
Copy link
Contributor Author

htrueman commented May 21, 2021 •
edited by github-actions bot

/test connector=source-sendgrid

🕑 source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/864339919
❌ source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/864339919

sherifnada
sherifnada approved these changes May 21, 2021
View reviewed changes
airbyte-integrations/connectors/source-sendgrid/source_sendgrid/streams.py
@@ -75,6 +76,29 @@ def next_page_token(self, response: requests.Response) -> Optional[Mapping[str,
return {"offset": self.offset}


class SendgridStreamIncrementalMixin(HttpStream):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nifty usage of mixin

@htrueman
Copy link
Contributor Author

htrueman commented May 25, 2021 •
edited by github-actions bot

/test connector=source-sendgrid

🕑 source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/874846144
✅ source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/874846144

@htrueman
Copy link
Contributor Author

htrueman commented May 25, 2021 •
edited by github-actions bot

/publish connector=connectors/source-sendgrid

🕑 connectors/source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/874914697
✅ connectors/source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/874914697

@htrueman htrueman merged commit 70f5dba into master May 25, 2021
@htrueman htrueman deleted the htrueman/source-sendgrid-support-incremental-sync branch May 25, 2021 12:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@keu keu keu approved these changes

@sherifnada sherifnada sherifnada approved these changes

@Zirochkaa Zirochkaa Awaiting requested review from Zirochkaa

@yevhenii-ldv yevhenii-ldv Awaiting requested review from yevhenii-ldv

@po3na4skld po3na4skld Awaiting requested review from po3na4skld

@vovavovavovavova vovavovavovavova Awaiting requested review from vovavovavovavova

@vitaliizazmic vitaliizazmic Awaiting requested review from vitaliizazmic

@ChristopheDuong ChristopheDuong Awaiting requested review from ChristopheDuong

Assignees

@htrueman htrueman

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support incremental sync in Sendgrid
3 participants
@htrueman @keu @sherifnada