TDL-12581: Increase tap tester coverage

[hpatel41]

Description of change

TDL-12581: Added integration test suite for:

• Test Sync Canary
• Test Discovery Mode (already exists but may be incomplete)
• Test Start Date
• Test Pagination (for "deals" stream)

Some of the highlighted changes as part of this PR:

• Bump pendulum 2.0.4 to 2.1.2 ( As with lower pendulum version, Sync mode was not running in CCI )
• Added Start date Key in the test as User and Deal Product result will be impacted by start date ( So mandatory to consider those in Start date integration-test ) but not impacted by states.
• Added Replication method in each stream. Below is the Mapping.
• Removed Replication Key for FULL_TABLE stream as it has no meaning.

Recent ( All are INCREMENTAL as sine_timestamp is mandatory param )

files
deals
activities
persons
products
organizations
notes

PipedriveIterStream
dealflow - It calls update_endpoint which calls get_deal_ids in which we run the loop to take data after updated_field so INCREMENTAL
Others ( Full_Table )

activity_types
currency
pipelines
stages
filters
users - It is a recent stream but replication_field is removed hence start date will be affected but state not hence FULL_TABLE
deal_products - It calls update_endpoint which calls get_deal_ids in which we run the loop to take data after start date hence start date will be affected but state not hence FULL_TABLE

Manual QA steps

• CCI build is passing.
• Discovery and Sync mode is running perfectly.
• Collected data for each stream with a different start_date. - Getting result as mentioned above

Risks

Rollback steps

• revert this branch

[KAllan357]

So tables like these had a state_field but were operating full_table anyway?

[dbshah1212]

Yes @KAllan357, Please find the doc link which shows all the streams having similar behavior in the tap https://talend365-my.sharepoint.com/:w:/g/personal/kpanchal_talend_com/EV4SU6EM5khDmDobC-hIXc4Bmv6cx-mx7NCOVLeCkS1HXg?e=B4Pohm