TDL-19486 Add limit clause to core stream queries #68

prijendev · 2022-06-17T08:48:23Z

Description of change

Added support of configurable page limit parameter into the tap.
LIMIT parameter is added for all core streams except the below streams,
- call_details
- campaign_lables
- ad_group_criterion
- campaign_criterion
LIMIT clauses in the ad_group_criterion and campaign_criterion(stream which has composite primary keys) may result in the infinite loop.
- For example, the limit is 10. campaign_criterion stream has a total of 20 records with campaign_id = 1 . So, in the first call, the tap retrieves 10 records, and the next time query would look like the below,
  WHERE campaign_id >= 1
- Now, the tap will again fetch records with campaign_id = 1. That's why we should not pass the LIMIT clause in the query of these streams.
- We can not do queries like WHERE pk_1 > 1 OR (pk_1 = 1 and pk_2 > 1) because google ads do not support OR parameter. We had a slack conversation about the same.
Added unit test case to validate different values of the query_limit parameter
Added integration test case.

QA steps

automated tests passing
manual qa steps passing

Risks

Dependent on this connections-service migration because testing expects a configurable query_limit to be accepted in config

Rollback steps

revert this branch

tap_google_ads/sync.py

tap_google_ads/streams.py

tap_google_ads/sync.py

tests/test_google_ads_page_limit.py

kspeer825 · 2022-06-21T14:28:16Z

tests/unittests/test_query_limit_param.py

+        "limit": value
+        }
+
+class TestQueryLimitParam(unittest.TestCase):


These are good test cases.

…eries

* Bump to v1.0.0, update changelog (singer-io#45) * Qa/future testing (singer-io#48) * documented reasons for untested streams * list high level scenarios for manual qa checks * remove stitch specific pr template Co-authored-by: kspeer <kspeer@stitchdata.com> * Add call_details stream (singer-io#49) * Add call_view core stream, filter non-attribute fields from core streams * Change `call_view` stream name to `call_details` * Make pylint happy * Fix integration tests * Remove campaign_id from foreign keys, split call_details foreign keys * Update tests to exclude call_details as needed * Add context around excluding call_details from tests * Remove outdated TODO related to addition of call_details stream. Co-authored-by: dsprayberry <28106103+dsprayberry@users.noreply.github.com> * Add core LABELS streams and campaign.labels fields to relevant reports (singer-io#53) * Add core LABELS streams and campaign.labels fields to relevant reports * Update test exclusions to exclude new core streams. * Update discover to include campaign_label in reports list * Update foreign_key expected metadata to include attributed resource foreign_keys * Update tests to re-include campaign_labels and labels as we now have test data. * Handles case where state does not have currently_syncing (singer-io#54) * Handles case where state does not have currently_syncing * tap-tester test added * takeout unused imports in test Co-authored-by: kspeer <kspeer@stitchdata.com> * Version bump and changelog entry (singer-io#52) * Version bump and changelog entry * Update to include PR 53. * Add PR 54 to changelog * Implement Automatic Keys (singer-io#55) * Set geographic_view.location_type as automatic to account for reporting discrepancies * Update Automatic Fields test happy / error paths * Update base to include automatic_keys metadata for use in tests * WIP Add Automatic Report Fields * Pass automatic_keys to BaseStream; use automatic_keys for inclusion * Fix bad field name in streams; Start updating tests * Rename function to match base.py * Add closing brace -_- * Fix tests; rename ad_group_criterion_criterion_id in automatic_keys; reorder test metadata; add campaign.id to campaign_audience_performance_report * Rename report_field_parts to split_report_field * Update transform_keys to raise ad_group_ad.ad fields * Update happy path streams_to_test to exclude streams with no data * Remove field name change for ad_performance_report stream * Accept Andy's Suggestion Co-authored-by: Andy Lu <andy@stitchdata.com> * Reverting Andy's change because it affects core streams * Explicitly install grpcio-status to avoid from_call attribute errors * Update setup.py with docs explaining required but unused dep Co-authored-by: Andy Lu <andy@stitchdata.com> Co-authored-by: Arthur Gorka <azgorka@gmail.com> * Cleanup (singer-io#56) * Remove useless function * Rename "REPORTS" to "STREAMS" for accuracy / readability * Version bump for PRs 56 and 55 (singer-io#57) * Version bump for PRs 56 and 55 * Update to exclude ad_group_ad change for ad_performance_report * [Feature] Add more core streams (singer-io#58) * WIP add new core streams * Set geographic_view.location_type as automatic to account for reporting discrepancies * Update Automatic Fields test happy / error paths * Update base to include automatic_keys metadata for use in tests * WIP Add Automatic Report Fields * Pass automatic_keys to BaseStream; use automatic_keys for inclusion * Fix bad field name in streams; Start updating tests * Rename function to match base.py * Add closing brace -_- * Fix tests; rename ad_group_criterion_criterion_id in automatic_keys; reorder test metadata; add campaign.id to campaign_audience_performance_report * Rename report_field_parts to split_report_field * Update transform_keys to raise ad_group_ad.ad fields * Update happy path streams_to_test to exclude streams with no data * WIP remove IPDB and try except * Fix report_definition typo and add handling for user_interest_id field * Remove field name change for ad_performance_report stream * Accept Andy's Suggestion Co-authored-by: Andy Lu <andy@stitchdata.com> * Reverting Andy's change because it affects core streams * Explicitly install grpcio-status to avoid from_call attribute errors * Remove ipdb -_- * Update setup.py with docs explaining required but unused dep * WIP w/ failing tests and attributed_resource foreign_keys * WIP with failing tests; remove extraneous attributed_resoruce automatic fields * Create UserInterestStream class to handle its edge case * Add transform_keys to UserInterestStream class; start test updates * Rename obj to json_message; rename variables accordingly; fix UserInterestStream transform_keys * update auto fields test to account for compound pks * Fix campaigns typo * Remove exclusion of feed and feed items from sync canary test Co-authored-by: Andy Lu <andy@stitchdata.com> Co-authored-by: Arthur Gorka <azgorka@gmail.com> Co-authored-by: kspeer <kspeer@stitchdata.com> Co-authored-by: atribed <agorka@talend.com> * V1.3.0 (singer-io#59) * Version bump for PRs 56 and 55 * Update to exclude ad_group_ad change for ad_performance_report * Version bump for 1.3.0 Co-authored-by: Arthur Gorka <azgorka@gmail.com> * Qa/fix build notification (singer-io#65) * fix slack notif for build * remove click_performance_report from tests * remove click_performance_report from sync canary test * run only streams that are untested in canary sync * just skip canaray test * put back the assert in the skipped test Co-authored-by: kspeer <kspeer@stitchdata.com> * Crest master (singer-io#66) * Tdl 19235 handle uncaught exceptions (singer-io#61) * Added backoff for 5xx, 429 and ReadTimeout errors. * Resolved pylint error. * Updated comments in the unittest cases. * Updated error handling. * TDL-18749 Implement interruptible full table streams. (singer-io#60) * Implemented interruptible full table streams. * Resolved pylint error * Resolved error in full table sync test case. * Updated config.yml to pass cci * Updated query building logic. * Updated integration test case. * Resolved review comments. * Resolved comments. * Implemeted logic to skip the duplicate records. * Resolved unittest case error. * Resolved pylint error * Resolved integration test case error * Added empty filter param for call_details and campaign_label stream. * Added unit test cases for should_sync method. * Revert "Implemeted logic to skip the duplicate records." This reverts commit cd06e11. * Added logger message for debugging purpose * Updated integration test case. * Replaced .format with f string. * Updated comment in integration test. Co-authored-by: KrishnanG <kgurusamy@talend.com> * Bump version (singer-io#67) Co-authored-by: KrishnanG <kgurusamy@talend.com> * TDL-18524 updated readme and added sample config (singer-io#51) * updated readme and added sample config * updated endpoints * add new streams * resolved PR comments * fixed a typo * added the * Add timeout parameter to gas.search (singer-io#64) * Add timeout parameter to search gas.search * Increase timeout to 15 minutes for safety. * Add get_request_timeout function, add config to make_request as needed * Update on_giveup to raise specific exception text for timeoutexception class; add unit test * Make Pylint happy take 1 * Update make_request signature in unittests. * Another Unittest update * More unittest fixes for signature * Add default config param value for ease of implementation in future tests * Fix pylint dangerous-defaul-value * Fix stupid error * Version bump and changelog (singer-io#70) * TDL-19486 Add limit clause to core stream queries (singer-io#68) * Initial commit for add page limit. * Added limit parameter in sync method of ReportStream class. * Fixed issue for call_details stream. * Resolved unit test case error. * Added test cases. * Updated code comments. * Fixed keyerror issue. * Updated default query limit. * Updated pagination test case. * Modify config name to be more explicit. * Update comment for accuracy. * Update property name in base and propogate name change to tests * Exclude feed from streams_to_test because of lack of data * Committing stuff from main that should have already been in Co-authored-by: dsprayberry <28106103+dsprayberry@users.noreply.github.com> * Version Bump and Changelog Update (singer-io#72) * Reintroduce unintentionally removed files. (singer-io#73) * add schemaless * fix schemaless config * update login_customer_ids * fix missing key from dict * add google ads api version in config add log to show the version * use version from config * use version global Co-authored-by: bryantgray <bgray@talend.com> Co-authored-by: Kyle Speer <54034650+kspeer825@users.noreply.github.com> Co-authored-by: kspeer <kspeer@stitchdata.com> Co-authored-by: dsprayberry <28106103+dsprayberry@users.noreply.github.com> Co-authored-by: Andy Lu <andy@stitchdata.com> Co-authored-by: Arthur Gorka <azgorka@gmail.com> Co-authored-by: atribed <agorka@talend.com> Co-authored-by: Prijen Khokhani <88327452+prijendev@users.noreply.github.com> Co-authored-by: KrishnanG <kgurusamy@talend.com> Co-authored-by: KrisPersonal <66801357+KrisPersonal@users.noreply.github.com> Co-authored-by: namrata270998 <75604662+namrata270998@users.noreply.github.com> Co-authored-by: bbaltagi-dtsl <bilal.baltagi@sendinblue.com>

Initial commit for add page limit.

c4c8252

prijendev changed the title ~~Initial commit for add page limit.~~ Add limit clause to core stream queries Jun 17, 2022

prijendev added 3 commits June 17, 2022 14:25

Added limit parameter in sync method of ReportStream class.

e75a1a4

Fixed issue for call_details stream.

be91c5c

Resolved unit test case error.

e4a6bc5

NevilParikh14 reviewed Jun 17, 2022

View reviewed changes

tap_google_ads/sync.py Outdated Show resolved Hide resolved

prijendev added 2 commits June 17, 2022 16:51

Added test cases.

a108a8b

Updated code comments.

8aa6493

prijendev requested a review from NevilParikh14 June 17, 2022 12:06

Fixed keyerror issue.

68d563d

prijendev requested review from dbshah1212 and savan-chovatiya June 17, 2022 12:32

prijendev changed the title ~~Add limit clause to core stream queries~~ TDL-19486 Add limit clause to core stream queries Jun 17, 2022

savan-chovatiya approved these changes Jun 17, 2022

View reviewed changes

prijendev requested review from KrisPersonal and kspeer825 June 17, 2022 13:54

dsprayberry requested changes Jun 17, 2022

View reviewed changes

tap_google_ads/streams.py Outdated Show resolved Hide resolved

tap_google_ads/sync.py Outdated Show resolved Hide resolved

tap_google_ads/sync.py Outdated Show resolved Hide resolved

tap_google_ads/sync.py Outdated Show resolved Hide resolved

Updated default query limit.

e4932f8

prijendev requested a review from dsprayberry June 20, 2022 10:22

dsprayberry approved these changes Jun 21, 2022

View reviewed changes

kspeer825 suggested changes Jun 21, 2022

View reviewed changes

Updated pagination test case.

f0b9eed

prijendev requested a review from kspeer825 June 22, 2022 13:00

kspeer825 approved these changes Jun 24, 2022

View reviewed changes

dsprayberry added 4 commits June 28, 2022 15:18

Modify config name to be more explicit.

7c65153

Update comment for accuracy.

820b8ee

Update property name in base and propogate name change to tests

ecd283a

Exclude feed from streams_to_test because of lack of data

f3bf6c0

dsprayberry mentioned this pull request Jun 30, 2022

Version Bump and Changelog Update #72

Merged

2 tasks

bryantgray approved these changes Jul 1, 2022

View reviewed changes

dsprayberry added 2 commits July 1, 2022 14:02

Merge branch 'main' into TDL-19486-add-limit-clause-to-core-stream-qu…

17478c3

…eries

Committing stuff from main that should have already been in

2cb0dfc

dsprayberry merged commit 2b9d9d2 into main Jul 5, 2022

dsprayberry deleted the TDL-19486-add-limit-clause-to-core-stream-queries branch July 5, 2022 13:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TDL-19486 Add limit clause to core stream queries #68

TDL-19486 Add limit clause to core stream queries #68

prijendev commented Jun 17, 2022 •

edited by dsprayberry

Loading

kspeer825 Jun 21, 2022

TDL-19486 Add limit clause to core stream queries #68

TDL-19486 Add limit clause to core stream queries #68

Conversation

prijendev commented Jun 17, 2022 • edited by dsprayberry Loading

Description of change

QA steps

Risks

Rollback steps

kspeer825 Jun 21, 2022

Choose a reason for hiding this comment

prijendev commented Jun 17, 2022 •

edited by dsprayberry

Loading