Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TDL-12581: Increase tap tester coverage #90

Merged
merged 55 commits into from
May 12, 2021

Conversation

hpatel41
Copy link
Contributor

@hpatel41 hpatel41 commented Apr 21, 2021

Description of change

TDL-12581: Added integration test suite for:

  • Test Sync Canary
  • Test Discovery Mode (already exists but may be incomplete)
  • Test Start Date
  • Test Pagination (for "deals" stream)

Some of the highlighted changes as part of this PR:

  • Bump pendulum 2.0.4 to 2.1.2 ( As with lower pendulum version, Sync mode was not running in CCI )
  • Added Start date Key in the test as User and Deal Product result will be impacted by start date ( So mandatory to consider those in Start date integration-test ) but not impacted by states.
  • Added Replication method in each stream. Below is the Mapping.
  • Removed Replication Key for FULL_TABLE stream as it has no meaning.

Recent ( All are INCREMENTAL as sine_timestamp is mandatory param )

files
deals
activities
persons
products
organizations
notes

PipedriveIterStream
dealflow - It calls update_endpoint which calls get_deal_ids in which we run the loop to take data after updated_field so INCREMENTAL
Others ( Full_Table )

activity_types
currency
pipelines
stages
filters
users - It is a recent stream but replication_field is removed hence start date will be affected but state not hence FULL_TABLE
deal_products - It calls update_endpoint which calls get_deal_ids in which we run the loop to take data after start date hence start date will be affected but state not hence FULL_TABLE

Manual QA steps

  • CCI build is passing.
  • Discovery and Sync mode is running perfectly.
  • Collected data for each stream with a different start_date. - Getting result as mentioned above

Risks

Rollback steps

  • revert this branch

@hpatel41 hpatel41 changed the title Tdl 12581 increase tap tester coverage TDL-12581: Increase tap tester coverage Apr 21, 2021
@@ -5,4 +5,5 @@ class PipelinesStream(PipedriveStream):
endpoint = 'pipelines'
schema = 'pipelines'
key_properties = ['id', ]
state_field = 'update_time'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So tables like these had a state_field but were operating full_table anyway?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes @KAllan357, Please find the doc link which shows all the streams having similar behavior in the tap https://talend365-my.sharepoint.com/:w:/g/personal/kpanchal_talend_com/EV4SU6EM5khDmDobC-hIXc4Bmv6cx-mx7NCOVLeCkS1HXg?e=B4Pohm

@KAllan357 KAllan357 merged commit 15dd61c into master May 12, 2021
@KAllan357 KAllan357 deleted the TDL-12581-increase-tap-tester-coverage branch May 12, 2021 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants