Source Pipedrive: handle missed cursor_field #24282

grubberr · 2023-03-21T12:07:54Z

What

Issue: #16376
Issue: https://github.com/airbytehq/alpha-beta-issues/issues/1002
Issue: #23687

Increase Users.page_size = 500 for Users stream. it seems pagination does not work for Users stream.
We cannot reproduce the issue on airbyte testing data. Temporary solution until the more better solution will be found.
Fix issue: Source Pipedrive: stuck reading Users records #23687
Skip re-evaluation state if cursor_field is missed in record.

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2023-03-21T12:15:14Z

/test connector=connectors/source-pipedrive

grubberr · 2023-03-21T12:24:13Z

/test connector=connectors/source-pipedrive

grubberr · 2023-03-21T12:32:05Z

/test connector=connectors/source-pipedrive

grubberr · 2023-03-21T12:40:44Z

/test connector=connectors/source-pipedrive

🕑 connectors/source-pipedrive https://github.com/airbytehq/airbyte/actions/runs/4479118977
✅ connectors/source-pipedrive https://github.com/airbytehq/airbyte/actions/runs/4479118977
Python tests coverage:

Name                           Stmts   Miss  Cover
--------------------------------------------------
source_pipedrive/auth.py           8      0   100%
source_pipedrive/__init__.py       2      0   100%
source_pipedrive/source.py        38      2    95%
source_pipedrive/streams.py      102     15    85%
--------------------------------------------------
TOTAL                            150     17    89%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:509: The previous and actual discovered catalogs are identical.
================== 36 passed, 2 skipped in 133.30s (0:02:13) ===================

lazebnyi · 2023-03-21T14:56:42Z

airbyte-integrations/connectors/source-pipedrive/source_pipedrive/streams.py

+        updated_state = latest_record.get(self.cursor_field)
+        if updated_state:
+            stream_state_value = current_stream_state.get(self.cursor_field)
+            if stream_state_value:
+                updated_state = max(updated_state, stream_state_value)
+            current_stream_state[self.cursor_field] = updated_state
+        return current_stream_state


According to spec.json replication_start_date is required and all cursor fields is date fields. So, we can used it as default cursor value. Also please check request_params method (line 62).

Suggested change

updated_state = latest_record.get(self.cursor_field)

if updated_state:

stream_state_value = current_stream_state.get(self.cursor_field)

if stream_state_value:

updated_state = max(updated_state, stream_state_value)

current_stream_state[self.cursor_field] = updated_state

return current_stream_state

updated_state = latest_record.get(self.cursor_field, self._replication_start_date)

stream_state_value = current_stream_state.get(self.cursor_field, self._replication_start_date)

current_stream_state[self.cursor_field] = max(updated_state, stream_state_value)

return current_stream_state

@lazebnyi
I have switched replication_start_date as default if cursor field is missed in record

about line 62
there is if condition because in "full refresh" streams self._replication_start_date still can be None.

replication_start_date is required, so I don't see any situations where it can be empty during sync.

we pass only to incremental stream like this

https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-pipedrive/source_pipedrive/source.py#L63

we don't pass to full_refresh streams like this:

https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-pipedrive/source_pipedrive/source.py#L62

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2023-03-21T15:38:51Z

/test connector=connectors/source-pipedrive

🕑 connectors/source-pipedrive https://github.com/airbytehq/airbyte/actions/runs/4480972065
✅ connectors/source-pipedrive https://github.com/airbytehq/airbyte/actions/runs/4480972065
Python tests coverage:

Name                           Stmts   Miss  Cover
--------------------------------------------------
source_pipedrive/auth.py           8      0   100%
source_pipedrive/__init__.py       2      0   100%
source_pipedrive/source.py        38      2    95%
source_pipedrive/streams.py      101     15    85%
--------------------------------------------------
TOTAL                            149     17    89%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:509: The previous and actual discovered catalogs are identical.
================== 36 passed, 2 skipped in 142.37s (0:02:22) ===================

airbyte-integrations/connectors/source-pipedrive/source_pipedrive/streams.py

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2023-03-21T19:06:46Z

/test connector=connectors/source-pipedrive

🕑 connectors/source-pipedrive https://github.com/airbytehq/airbyte/actions/runs/4482890509
✅ connectors/source-pipedrive https://github.com/airbytehq/airbyte/actions/runs/4482890509
Python tests coverage:

Name                           Stmts   Miss  Cover
--------------------------------------------------
source_pipedrive/auth.py           8      0   100%
source_pipedrive/__init__.py       2      0   100%
source_pipedrive/source.py        38      2    95%
source_pipedrive/streams.py       98     15    85%
--------------------------------------------------
TOTAL                            146     17    88%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:509: The previous and actual discovered catalogs are identical.
================== 36 passed, 2 skipped in 141.14s (0:02:21) ===================

grubberr · 2023-03-21T20:05:36Z

/publish connector=connectors/source-pipedrive

🕑 Publishing the following connectors:
connectors/source-pipedrive
https://github.com/airbytehq/airbyte/actions/runs/4483345849

Connector	Did it publish?	Were definitions generated?
connectors/source-pipedrive	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> (cherry picked from commit 164d1c1)

* Source Pipedrive: handle missed cursor_field (airbytehq#24282) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> (cherry picked from commit 164d1c1) * chore: remove oauth for pipedrive --------- Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>

improve get_updated_state

9cbfb03

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr self-assigned this Mar 21, 2023

octavia-squidington-iii added area/connectors Connector related issues connectors/source/pipedrive labels Mar 21, 2023

grubberr requested review from lazebnyi and davydov-d March 21, 2023 13:43

lazebnyi requested changes Mar 21, 2023

View reviewed changes

grubberr added 2 commits March 21, 2023 15:14

Merge branch 'master' into grubberr/16376-source-pipedrive

9e08c62

self._replication_start_date added default for get_updated_state

36be43e

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr requested a review from lazebnyi March 21, 2023 16:05

lazebnyi requested changes Mar 21, 2023

View reviewed changes

airbyte-integrations/connectors/source-pipedrive/source_pipedrive/streams.py Outdated Show resolved Hide resolved

grubberr requested a review from lazebnyi March 21, 2023 18:45

grubberr added 4 commits March 21, 2023 18:47

Merge branch 'master' into grubberr/16376-source-pipedrive

40e5d23

improve get_updated_state

40d4e10

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

bump 0.1.17

9ccc2c1

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

updated pipedrive.md

bc2836b

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Mar 21, 2023

lazebnyi approved these changes Mar 21, 2023

View reviewed changes

auto-bump connector version

0706f49

grubberr merged commit 164d1c1 into master Mar 21, 2023
20 of 21 checks passed

grubberr deleted the grubberr/16376-source-pipedrive branch March 21, 2023 20:49

This was referenced Mar 21, 2023

Source Pipedrive: Incremental sync KeyError: 'update_time' #16376

Closed

Source Pipedrive: stuck reading Users records #23687

Closed

erohmensing pushed a commit that referenced this pull request Mar 22, 2023

Source Pipedrive: handle missed cursor_field (#24282)

8527192

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

erohmensing pushed a commit that referenced this pull request Mar 22, 2023

Source Pipedrive: handle missed cursor_field (#24282)

f198289

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

am6010 pushed a commit to rudderlabs/airbyte that referenced this pull request Mar 27, 2023

Source Pipedrive: handle missed cursor_field (airbytehq#24282)

ea600f3

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> (cherry picked from commit 164d1c1)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Pipedrive: handle missed cursor_field #24282

Source Pipedrive: handle missed cursor_field #24282

grubberr commented Mar 21, 2023 •

edited

grubberr commented Mar 21, 2023

grubberr commented Mar 21, 2023

grubberr commented Mar 21, 2023

grubberr commented Mar 21, 2023 •

edited by github-actions bot

lazebnyi Mar 21, 2023

grubberr Mar 21, 2023

lazebnyi Mar 21, 2023

grubberr Mar 21, 2023

grubberr commented Mar 21, 2023 •

edited by github-actions bot

grubberr commented Mar 21, 2023 •

edited by github-actions bot

grubberr commented Mar 21, 2023 •

edited by github-actions bot

Source Pipedrive: handle missed cursor_field #24282

Source Pipedrive: handle missed cursor_field #24282

Conversation

grubberr commented Mar 21, 2023 • edited

What

Community member or Airbyter

Airbyter

grubberr commented Mar 21, 2023

grubberr commented Mar 21, 2023

grubberr commented Mar 21, 2023

grubberr commented Mar 21, 2023 • edited by github-actions bot

Build Passed

lazebnyi Mar 21, 2023

Choose a reason for hiding this comment

grubberr Mar 21, 2023

Choose a reason for hiding this comment

lazebnyi Mar 21, 2023

Choose a reason for hiding this comment

grubberr Mar 21, 2023

Choose a reason for hiding this comment

grubberr commented Mar 21, 2023 • edited by github-actions bot

Build Passed

grubberr commented Mar 21, 2023 • edited by github-actions bot

Build Passed

grubberr commented Mar 21, 2023 • edited by github-actions bot

grubberr commented Mar 21, 2023 •

edited

grubberr commented Mar 21, 2023 •

edited by github-actions bot

grubberr commented Mar 21, 2023 •

edited by github-actions bot

grubberr commented Mar 21, 2023 •

edited by github-actions bot

grubberr commented Mar 21, 2023 •

edited by github-actions bot