✨ Source Mailchimp: Add optional start_date to config #32852

ChristoGrab · 2023-11-27T20:34:57Z

What

Adds an optional start_date to Source Mailchimp's config, to resolve customer request in #20814

How

If a start_date is provided by a user via config, incremental streams will compare it against the value found in the current stream slice's state, to determine which to use as a filter in the query params. If only one of these values exists, it will be used by default.
If no start_date is set by a user in the config and the slice's state is empty (ie first sync), the filtering query param is not added to the request, and all available data is fetched from the API.
The API endpoints for two of the existing incremental streams, Segment Members and Unsubscribes, do not support filtering by the cursor value in query params, and will always return all available data. For these streams, the same filtering logic takes place, except this occurs during the parsing of the response, before the records are read.

User Impact

No impact on current customers. The start_date in config is an optional input and defaults to None if left empty. Therefore, for users with existing connections, the currently existing logic resumes, whereby the connector always fetches and reads all available data during first sync.

…32218)

…stamp_without_timezone (#32595)

…fic_type` field to AdGroups stream (#32747)

… ad accounts during `check` (#32731)

… expected records, migrate base image (#32718) Co-authored-by: pnilan <pnilan@users.noreply.github.com>

)

vercel · 2023-11-27T20:37:00Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 20, 2023 6:47pm

github-actions · 2023-11-27T20:37:15Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

…bytehq/airbyte into christo/mailchimp-start-date

lazebnyi · 2023-12-20T10:32:33Z

airbyte-integrations/connectors/source-mailchimp/source_mailchimp/streams.py

+        if start_date_parsed and state_date_parsed:
+            return max(start_date_parsed, state_date_parsed)
+        elif state_date_parsed or start_date_parsed:
+            return state_date_parsed or start_date_parsed
+        else:
+            return None


Suggested change

if start_date_parsed and state_date_parsed:

return max(start_date_parsed, state_date_parsed)

elif state_date_parsed or start_date_parsed:

return state_date_parsed or start_date_parsed

else:

return None

# Return the maximum of the two dates if both are present, otherwise return whichever is present, or None

if start_date_parsed or state_date_parsed:

return max(filter(None, [start_date_parsed, state_date_parsed]), default=None)

lazebnyi · 2023-12-20T10:57:07Z

airbyte-integrations/connectors/source-mailchimp/source_mailchimp/streams.py

@@ -157,7 +181,11 @@ def stream_slices(self, stream_state: Mapping[str, Any] = None, **kwargs) -> Ite
        stream_state = stream_state or {}
        parent = Lists(authenticator=self.authenticator).read_records(sync_mode=SyncMode.full_refresh)
        for slice in parent:
-            yield {"list_id": slice["id"]}
+            slice_ = {"list_id": slice["id"]}


I believe we can rename slice to parent_record and slice_ to slice

Good call, updated

lazebnyi · 2023-12-20T10:58:41Z

airbyte-integrations/connectors/source-mailchimp/source_mailchimp/streams.py

+            if filter_date is None or record_cursor_value >= filter_date:
                yield record


this is can be separate filtration function

check Hubspot connector as example

Thanks for the pointer 👍 A separate method, filter_old_records, is now declared in the base MailchimpIncrementalStream class and invoked by streams which do not support the server-side filter.

lazebnyi

Nice!

ChristoGrab and others added 19 commits November 16, 2023 08:12

add start date to config

47a6a5a

Source Hubspot: fix expected records (#32645)

b611729

Source Github: fix expected records (#32644)

a9cc2b9

✨ Source Mailchimp: Add Interests, InterestCategories, Tags streams (#…

e74bbde

…32218)

🐛 Source Pinterest: Fix backoff waiting time (#32672)

918e64c

🚨🚨🐛 Source Pinterest: Update date-time fields with airbyte_type: time…

0f28144

…stamp_without_timezone (#32595)

remove bq+snowflake from legacy normalization docs (#32708)

39ac79c

🐛 Source Pinterest: Fix Advertiser stream names (#32734)

667be92

✨ Source Pinterest: Update docs and spec; add missing `placement_traf…

059ca2f

…fic_type` field to AdGroups stream (#32747)

🐛 Source Facebook Marketing: Removed validation that blocked personal…

d81ae88

… ad accounts during `check` (#32731)

Docs: Add permissions to prereqs in Source Facebook Marketing (#32653)

74d359d

✨ Source Mailchimp: Implement SegmentMembers stream (#32782)

3e69945

Source My Hours: Update CDK (#32680)

4cdfc7a

add logic for use of start_date in incremental streams

428e1b3

Source Cart: Update CDK to Latest Version (#32705)

b5d0ede

Source Twilio: Increase test coverage, fix parse_response bug, update…

86848a6

… expected records, migrate base image (#32718) Co-authored-by: pnilan <pnilan@users.noreply.github.com>

Source Sendgrid: Increase Test Coverage, Update Expected Records (#32742

1fda1df

)

fix method and add unit test

941d937

version bump

d17e251

octavia-squidington-iii added area/connectors Connector related issues connectors/source/mailchimp labels Nov 27, 2023

small edits

b2a546e

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Nov 27, 2023

ChristoGrab and others added 4 commits November 27, 2023 20:45

Automated Commit - Formatting Changes

1a0bb33

✨Source Amazon Seller Partner: multiple updates (#32833)

a660279

add note to docs

00a896b

Merge branch 'christo/mailchimp-start-date' of https://github.com/air…

05c5acb

…bytehq/airbyte into christo/mailchimp-start-date

ChristoGrab marked this pull request as ready for review November 28, 2023 16:38

Automated Commit - Formatting Changes

a400070

ChristoGrab requested a review from a team as a code owner December 8, 2023 18:58

octavia-squidington-iii added connectors/destination/milvus connectors/source/hubspot connectors/source/amazon-seller-partner connectors/source/pinterest connectors/source/mongodb-v2 connectors/source/my-hours connectors/source/datadog connectors/source/klaus-api labels Dec 8, 2023

chore: fix merge conflict

0195913

vercel bot had a problem deploying to Preview December 8, 2023 19:12 Failure

Merge branch 'master' into christo/mailchimp-start-date

4814279

vercel bot deployed to Preview December 11, 2023 15:55 View deployment

Merge branch 'master' into christo/mailchimp-start-date

918388d

vercel bot deployed to Preview December 11, 2023 16:31 View deployment

ChristoGrab added 2 commits December 19, 2023 18:15

merge changes from update to 1.0.0

b7d30b7

Merge branch 'christo/mailchimp-start-date' of https://github.com/air…

2b4b74f

…bytehq/airbyte into christo/mailchimp-start-date

lazebnyi requested changes Dec 20, 2023

View reviewed changes

octavia-squidington-iv requested a review from a team December 20, 2023 10:59

ChristoGrab added 2 commits December 20, 2023 13:23

add separate method for client-side filter

f6953e3

chore: format fix

aa2b6de

vercel bot deployed to Preview December 20, 2023 18:47 View deployment

ChristoGrab requested a review from lazebnyi December 20, 2023 18:49

lazebnyi approved these changes Dec 20, 2023

View reviewed changes

ChristoGrab merged commit 8d33e38 into master Dec 20, 2023
26 checks passed

ChristoGrab deleted the christo/mailchimp-start-date branch December 20, 2023 21:59

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

✨ Source Mailchimp: Add optional start_date to config (airbytehq#32852)

7556171

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

✨ Source Mailchimp: Add optional start_date to config (airbytehq#32852)

47ae3d1

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

✨ Source Mailchimp: Add optional start_date to config (airbytehq#32852)

9b739f5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Source Mailchimp: Add optional start_date to config #32852

✨ Source Mailchimp: Add optional start_date to config #32852

ChristoGrab commented Nov 27, 2023 •

edited

Loading

vercel bot commented Nov 27, 2023 •

edited

Loading

github-actions bot commented Nov 27, 2023 •

edited by ChristoGrab

Loading

lazebnyi Dec 20, 2023

ChristoGrab Dec 20, 2023

lazebnyi Dec 20, 2023

ChristoGrab Dec 20, 2023

lazebnyi Dec 20, 2023

ChristoGrab Dec 20, 2023 •

edited

Loading

lazebnyi left a comment

		if filter_date is None or record_cursor_value >= filter_date:
		yield record

✨ Source Mailchimp: Add optional start_date to config #32852

✨ Source Mailchimp: Add optional start_date to config #32852

Conversation

ChristoGrab commented Nov 27, 2023 • edited Loading

What

How

User Impact

vercel bot commented Nov 27, 2023 • edited Loading

github-actions bot commented Nov 27, 2023 • edited by ChristoGrab Loading

Before Merging a Connector Pull Request

lazebnyi Dec 20, 2023

Choose a reason for hiding this comment

ChristoGrab Dec 20, 2023

Choose a reason for hiding this comment

lazebnyi Dec 20, 2023

Choose a reason for hiding this comment

ChristoGrab Dec 20, 2023

Choose a reason for hiding this comment

lazebnyi Dec 20, 2023

Choose a reason for hiding this comment

ChristoGrab Dec 20, 2023 • edited Loading

Choose a reason for hiding this comment

lazebnyi left a comment

Choose a reason for hiding this comment

ChristoGrab commented Nov 27, 2023 •

edited

Loading

vercel bot commented Nov 27, 2023 •

edited

Loading

github-actions bot commented Nov 27, 2023 •

edited by ChristoGrab

Loading

ChristoGrab Dec 20, 2023 •

edited

Loading