Sync record from start date for streams - campaigns, ad groups, ads #22
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RushiT0122
requested changes
Jun 19, 2023
tap_tiktok_ads/streams.py
Outdated
Comment on lines
325
to
329
| for date_batch in date_batches: | ||
| filter = {'create_start_time': strftime(date_batch['start_date'], FILTER_DATE_FORMAT), | ||
| 'create_end_time': strftime(date_batch['end_date'], FILTER_DATE_FORMAT)} | ||
| self.params['filtering'] = json.dumps(filter) | ||
| self.sync_pages(stream) |
There was a problem hiding this comment.
Let's say we are extracting campaigns stream and We start date is 2022-04-21 then we will miss such records.
id create_time modify_time
1 2022-04-18T08:32:17 2022-04-21T08:32:17
Please re-verify the logic.
There was a problem hiding this comment.
Removed those filters. As the API does not support querying using the replication key, therefore, pseudo-incremental (aka full table) logic is implemented. Now, Tap will only emit records with a replication key greater than our bookmark
RushiT0122
approved these changes
Jun 21, 2023
LesleyDing
pushed a commit
to vibeus/tap-tiktok-ads
that referenced
this pull request
Mar 25, 2024
…inger-io#22) * Sync record from start date for streams - campaigns, ad groups, ads * fix start date test * remove the query filter and do pseudo increment for streams * fix the unit test cases * remove unused code * setup and changelog update --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of change
Initially, records older than the start date were included in the target DB. To refine this, Looked at the API documentation. We cannot query via an updated_at or modified_at, only by create_time so we will be treating the stream as pseudo-incremental, aka full table but we only emit records with a replication key greater than our bookmark
Manual QA steps
Risks
Rollback steps