✨ Source Monday: Add incremental sync for Items stream #27944

tolik0 · 2023-07-04T12:34:20Z

What

Implements incremental sync for Items stream. Adds new stream: Activity logs.
#20950

How

This pull request implements incremental synchronization for the Items stream by implementing the approach described in the connected issue. The implementation follows these steps:

Initial Sync: All items are synced initially without relying on activity logs.
Subsequent Sync: The process involves fetching updates since a specified date from the Activity logs stream. The modified item IDs are extracted from the activity logs. Based on these extracted IDs, the corresponding items are selectively synced to ensure their data is up to date.

Following this process, the pull request achieves incremental synchronization by selectively updating only the modified items, as indicated by the activity logs.

source-monday test report (commit `abde7c925e`) - ❌

⏲️ Total pipeline duration: 01mn31s

Step	Result
Validate airbyte-integrations/connectors/source-monday/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-monday docker image for platform linux/x86_64	✅
Unit tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-monday test

octavia-squidington-iii · 2023-07-07T17:07:37Z

source-monday test report (commit `7b7c9abcf5`) - ❌

⏲️ Total pipeline duration: 19mn38s

Step	Result
Validate airbyte-integrations/connectors/source-monday/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Code format checks	❌
Connector package install	✅
Build source-monday docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-monday test

octavia-squidington-iii · 2023-07-07T18:27:45Z

source-monday test report (commit `16c5918f8b`) - ✅

⏲️ Total pipeline duration: 17mn48s

Step	Result
Validate airbyte-integrations/connectors/source-monday/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-monday docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-monday test

lazebnyi · 2023-07-10T10:36:07Z

airbyte-integrations/connectors/source-monday/acceptance-test-config.yml

-        # this change applies for all configs bellow.
+        #  changed type of id for assets in items and updates
+        #  change deprecated owner to owners in boards
+        # All changes described in corresponding pr


Please add link to PR and delete
# changed type of id for assets in items and updates
# change deprecated owner to owners in boards

please add link to PR instead number

airbyte-integrations/connectors/source-monday/source_monday/components.py

lazebnyi · 2023-07-10T11:24:34Z

airbyte-integrations/connectors/source-monday/source_monday/components.py

+    def close_slice(self, stream_slice: StreamSlice, most_recent_record: Optional[Record]) -> None:
+        max_dt = self._state if self.is_greater_than_or_equal(self._state, most_recent_record) else most_recent_record
+
+        if not max_dt:
+            return
+        self._state[self.cursor_field.eval(self.config)] = max_dt[self.cursor_field.eval(self.config)]
+
+        if self.parent_stream:
+            if self.parent_cursor_field not in most_recent_record:
+                date_str = most_recent_record[self.cursor_field.eval(self.config)]
+                parent_cursor = int(datetime.strptime(date_str, "%Y-%m-%dT%H:%M:%S%z").timestamp())
+            else:
+                parent_cursor = most_recent_record[self.parent_cursor_field]
+            self._state[self.parent_stream_name] = {self.parent_cursor_field: parent_cursor}


Suggested change

def close_slice(self, stream_slice: StreamSlice, most_recent_record: Optional[Record]) -> None:

max_dt = self._state if self.is_greater_than_or_equal(self._state, most_recent_record) else most_recent_record

if not max_dt:

return

self._state[self.cursor_field.eval(self.config)] = max_dt[self.cursor_field.eval(self.config)]

if self.parent_stream:

if self.parent_cursor_field not in most_recent_record:

date_str = most_recent_record[self.cursor_field.eval(self.config)]

parent_cursor = int(datetime.strptime(date_str, "%Y-%m-%dT%H:%M:%S%z").timestamp())

else:

parent_cursor = most_recent_record[self.parent_cursor_field]

self._state[self.parent_stream_name] = {self.parent_cursor_field: parent_cursor}

def close_slice(self, stream_slice: StreamSlice, last_record: Optional[Record]) -> None:

max_dt = self._state if self.is_greater_than_or_equal(self._state, most_recent_record) else most_recent_record

if not max_dt:

return

cursor_field_eval = self.cursor_field.eval(self.config)

self._state[cursor_field_eval] = max_dt[cursor_field_eval]

if self.parent_stream:

date_str = most_recent_record[self.cursor_field.eval(self.config)]

default_parent_cursor = int(datetime.strptime(date_str, "%Y-%m-%dT%H:%M:%S%z").timestamp())

parent_cursor = most_recent_record.get(self.parent_cursor_field, default_parent_cursor)

self._state[self.parent_stream_name] = {self.parent_cursor_field: parent_cursor}

airbyte-integrations/connectors/source-monday/source_monday/components.py

airbyte-integrations/connectors/source-monday/source_monday/extractor.py

octavia-squidington-iii · 2023-07-11T17:28:32Z

source-monday test report (commit `1746c57a70`) - ❌

⏲️ Total pipeline duration: 57.28s

Step	Result
Validate airbyte-integrations/connectors/source-monday/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-monday docker image for platform linux/x86_64	✅
Unit tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-monday test

octavia-squidington-iii · 2023-07-11T17:40:03Z

source-monday test report (commit `df87dd0970`) - ❌

⏲️ Total pipeline duration: 36.09s

Step	Result
Validate airbyte-integrations/connectors/source-monday/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-monday docker image for platform linux/x86_64	✅
Unit tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-monday test

octavia-squidington-iii · 2023-07-11T17:55:52Z

source-monday test report (commit `19cbfa94ca`) - ✅

⏲️ Total pipeline duration: 04mn08s

Step	Result
Validate airbyte-integrations/connectors/source-monday/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-monday docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-monday test

airbyte-integrations/connectors/source-monday/source_monday/extractor.py

airbyte-integrations/connectors/source-monday/acceptance-test-config.yml

…m-incremental-sync

octavia-squidington-iii · 2023-07-12T00:48:15Z

source-monday test report (commit `4c31c40767`) - ✅

⏲️ Total pipeline duration: 04mn14s

Step	Result
Validate airbyte-integrations/connectors/source-monday/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-monday docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-monday test

lazebnyi

LGTM! Minor change required.

airbyte-integrations/connectors/source-monday/acceptance-test-config.yml

octavia-squidington-iii · 2023-07-12T14:47:25Z

source-monday test report (commit `334138abe9`) - ✅

⏲️ Total pipeline duration: 04mn11s

Step	Result
Validate airbyte-integrations/connectors/source-monday/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-monday docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-monday test

…rbytehq#27944) * Add incremental sync for the Boards and Items streams using Activity logs --------- Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com> Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

tolik0 linked an issue Jul 4, 2023 that may be closed by this pull request

Source Monday.com: Incremental Sync #20950

Closed

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/monday labels Jul 4, 2023

tolik0 requested a review from bazarnov July 4, 2023 12:34

tolik0 self-assigned this Jul 5, 2023

bazarnov requested changes Jul 5, 2023

View reviewed changes

tolik0 force-pushed the tolik0/source-monday/20950-source-mondaycom-incremental-sync branch from c267655 to 0c95f12 Compare July 5, 2023 13:38

tolik0 changed the title ~~Source Monday: Add incremental sync~~ ✨ Source Monday: Add incremental sync for Items stream Jul 7, 2023

tolik0 marked this pull request as ready for review July 7, 2023 16:09

tolik0 requested a review from bazarnov July 7, 2023 16:10

tolik0 requested review from roman-yermilov-gl and lazebnyi July 7, 2023 16:58

bazarnov approved these changes Jul 8, 2023

View reviewed changes

lazebnyi requested changes Jul 10, 2023

View reviewed changes

bazarnov and others added 10 commits July 11, 2023 20:04

updated public docs

4296a39

Update monday.md

46648fc

Add incremental Items stream

a71f943

Update slicer classes to inherit from the Cursor class

b8f2ffc

Fix formatting

0393bce

Update docs

865f401

Fix new lines formatting

4ae5d61

Fix time formats

f63ceac

Fix docs

10b263c

Add incremental sync for Boards stream

4d4abc4

tolik0 force-pushed the tolik0/source-monday/20950-source-mondaycom-incremental-sync branch from 8402029 to 4d4abc4 Compare July 11, 2023 17:05

tolik0 added 2 commits July 11, 2023 20:13

Fix minor errors

1746c57

Add test_strictness_level high to acceptance tests

df87dd0

Fix unit tests

19cbfa9

tolik0 requested review from lazebnyi and bazarnov July 11, 2023 18:02

lazebnyi requested changes Jul 11, 2023

View reviewed changes

airbyte-integrations/connectors/source-monday/source_monday/extractor.py Outdated Show resolved Hide resolved

airbyte-integrations/connectors/source-monday/source_monday/extractor.py Outdated Show resolved Hide resolved

lazebnyi reviewed Jul 11, 2023

View reviewed changes

airbyte-integrations/connectors/source-monday/acceptance-test-config.yml Outdated Show resolved Hide resolved

tolik0 and others added 4 commits July 11, 2023 23:06

Fix formatting

b14802d

Minor fix

8398453

Merge branch 'master' into tolik0/source-monday/20950-source-mondayco…

fe2c759

…m-incremental-sync

Merge branch 'master' into tolik0/source-monday/20950-source-mondayco…

4c31c40

…m-incremental-sync

tolik0 requested a review from lazebnyi July 11, 2023 22:33

lazebnyi approved these changes Jul 12, 2023

View reviewed changes

airbyte-integrations/connectors/source-monday/acceptance-test-config.yml Outdated Show resolved Hide resolved

Minor fix

334138a

tolik0 merged commit 507ad69 into master Jul 13, 2023
22 of 23 checks passed

tolik0 deleted the tolik0/source-monday/20950-source-mondaycom-incremental-sync branch July 13, 2023 09:13

arsenlosenko mentioned this pull request Jul 13, 2023

🐛 Source Monday: update expected records #28198

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Source Monday: Add incremental sync for Items stream #27944

✨ Source Monday: Add incremental sync for Items stream #27944

tolik0 commented Jul 4, 2023 •

edited

github-actions bot commented Jul 4, 2023 •

edited by tolik0

bazarnov left a comment

octavia-squidington-iii commented Jul 7, 2023

octavia-squidington-iii commented Jul 7, 2023

octavia-squidington-iii commented Jul 7, 2023

lazebnyi Jul 10, 2023

tolik0 Jul 11, 2023

lazebnyi Jul 11, 2023

lazebnyi Jul 10, 2023

octavia-squidington-iii commented Jul 11, 2023

octavia-squidington-iii commented Jul 11, 2023

octavia-squidington-iii commented Jul 11, 2023

octavia-squidington-iii commented Jul 12, 2023

lazebnyi left a comment

octavia-squidington-iii commented Jul 12, 2023

✨ Source Monday: Add incremental sync for Items stream #27944

✨ Source Monday: Add incremental sync for Items stream #27944

Conversation

tolik0 commented Jul 4, 2023 • edited

What

How

Recommended reading order

github-actions bot commented Jul 4, 2023 • edited by tolik0

Before Merging a Connector Pull Request

bazarnov left a comment

Choose a reason for hiding this comment

octavia-squidington-iii commented Jul 7, 2023

source-monday test report (commit abde7c925e) - ❌

octavia-squidington-iii commented Jul 7, 2023

source-monday test report (commit 7b7c9abcf5) - ❌

octavia-squidington-iii commented Jul 7, 2023

source-monday test report (commit 16c5918f8b) - ✅

lazebnyi Jul 10, 2023

Choose a reason for hiding this comment

tolik0 Jul 11, 2023

Choose a reason for hiding this comment

lazebnyi Jul 11, 2023

Choose a reason for hiding this comment

lazebnyi Jul 10, 2023

Choose a reason for hiding this comment

octavia-squidington-iii commented Jul 11, 2023

source-monday test report (commit 1746c57a70) - ❌

octavia-squidington-iii commented Jul 11, 2023

source-monday test report (commit df87dd0970) - ❌

octavia-squidington-iii commented Jul 11, 2023

source-monday test report (commit 19cbfa94ca) - ✅

octavia-squidington-iii commented Jul 12, 2023

source-monday test report (commit 4c31c40767) - ✅

lazebnyi left a comment

Choose a reason for hiding this comment

octavia-squidington-iii commented Jul 12, 2023

source-monday test report (commit 334138abe9) - ✅

tolik0 commented Jul 4, 2023 •

edited

github-actions bot commented Jul 4, 2023 •

edited by tolik0

source-monday test report (commit `abde7c925e`) - ❌

source-monday test report (commit `7b7c9abcf5`) - ❌

source-monday test report (commit `16c5918f8b`) - ✅

source-monday test report (commit `1746c57a70`) - ❌

source-monday test report (commit `df87dd0970`) - ❌

source-monday test report (commit `19cbfa94ca`) - ✅

source-monday test report (commit `4c31c40767`) - ✅

source-monday test report (commit `334138abe9`) - ✅