[ISSUE #26607] zenloop migration #27243

maxi297 · 2023-06-12T14:43:52Z

What

Addresses #26607

How

Improving error message
Updating CATs
Adding change logs

🚨 User Impact 🚨

As mentioned in the change logs, the user will have to reset the sync and resync.

maxi297 · 2023-06-12T14:44:09Z

/test connector=connectors/source-zenloop local_cdk=1

🕑 connectors/source-zenloop https://github.com/airbytehq/airbyte/actions/runs/5245040522
❌ connectors/source-zenloop https://github.com/airbytehq/airbyte/actions/runs/5245040522
🐛 https://gradle.com/s/xztaazrt7sauy

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:598: The previous and actual discovered catalogs are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:695: This tests currently leads to too much failures. We need to fix the connectors at scale first.
======= 3 failed, 35 passed, 3 skipped, 41 warnings in 63.67s (0:01:03) ========

maxi297 · 2023-06-12T15:04:49Z

/test connector=connectors/source-zenloop local_cdk=1

🕑 connectors/source-zenloop https://github.com/airbytehq/airbyte/actions/runs/5245263874
❌ connectors/source-zenloop https://github.com/airbytehq/airbyte/actions/runs/5245263874
🐛 https://gradle.com/s/mrjypz6kprrbm

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:598: The previous and actual discovered catalogs are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:695: This tests currently leads to too much failures. We need to fix the connectors at scale first.
======= 2 failed, 36 passed, 3 skipped, 41 warnings in 74.13s (0:01:14) ========

maxi297 · 2023-06-12T19:30:53Z

/test connector=connectors/source-zenloop local_cdk=1

🕑 connectors/source-zenloop https://github.com/airbytehq/airbyte/actions/runs/5247740004
❌ connectors/source-zenloop https://github.com/airbytehq/airbyte/actions/runs/5247740004
🐛 https://gradle.com/s/bw2einrisozhk

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:598: The previous and actual discovered catalogs are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:695: This tests currently leads to too much failures. We need to fix the connectors at scale first.
======= 3 failed, 35 passed, 3 skipped, 41 warnings in 65.02s (0:01:05) ========

maxi297 · 2023-06-14T12:56:25Z

This is a breaking change but we don't have any zenloop users on cloud so we won't fill https://docs.google.com/document/d/1VYQggHbL_PN0dDDu7rCyzBLGRtX-R3cpwXaY8QxEgzw/edit#heading=h.r2xzw0kclodc

maxi297 · 2023-06-14T13:33:30Z

/test connector=connectors/source-zenloop local_cdk=1

🕑 connectors/source-zenloop https://github.com/airbytehq/airbyte/actions/runs/5267836110
✅ connectors/source-zenloop https://github.com/airbytehq/airbyte/actions/runs/5267836110
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:598: The previous and actual discovered catalogs are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:695: This tests currently leads to too much failures. We need to fix the connectors at scale first.
============ 38 passed, 3 skipped, 41 warnings in 77.25s (0:01:17) =============

…oop-migration-to-per-partition-cursor

alafanechere · 2023-06-14T16:56:35Z

I'm updating the PR to get fresh CI run. CF https://airbytehq-team.slack.com/archives/C02UF50V9HA/p1686758235881369

…oop-migration-to-per-partition-cursor

alafanechere · 2023-06-14T17:53:06Z

@maxi297 you don't get fresh CI because the target branch is not master. The target branch must be updated too.

maxi297 · 2023-06-14T18:33:21Z

@alafanechere Target branch has been updated. I'll merge this in the target branch to leverage the fresh CI stuff

…oop-migration-to-per-partition-cursor

octavia-squidington-iii · 2023-06-16T19:48:03Z

source-zenloop test report (commit `9f18ccb2db`) - ❌

⏲️ Total pipeline duration: 148 seconds

Step	Result
Validate airbyte-integrations/connectors/source-zenloop/metadata.yaml	✅
Connector version semver check.	✅
QA checks	✅
Connector package install	✅
Build source-zenloop docker image for platform linux/x86_64	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-zenloop test

This reverts commit 5c6f19b.

This reverts commit e363fd6.

* [ISSUE #26581] per partition cursor * [ISSUE #26581] format * [ISSUE #26581] clean up state management * [ISSUE #26581] improving Hashabledict * [ISSUE #26581] format cdk * [ISSUE #26581] fix tests * [ISSUE #26581] code review from girarda * Retrigger pipeline * [ISSUE #26581] code review * Automated Commit - Formatting Changes * [ISSUE #26581] validation overlapping keys * [ISSUE #26581] add typing * [ISSUE #26581] code review * [ISSUE #26607] zenloop migration (#27243) * [ISSUE #26607] zenloop migration implementation without tests * [ISSUE #26607] zenloop migration adding edge cases * [ISSUE #26607] add cursor field for state * [ISSUE #26607] update abnormal state * [ISSUE #26607] ensure default state * [ISSUE #26607] updating CATs state * [ISSUE #26607] revert migrating cursor * [ISSUE #26607] remove default cursor value * [ISSUE #26607] improve error message * [ISSUE #26607] changelog --------- Co-authored-by: Augustin <augustin@airbyte.io> * 🤖 Auto format source-zenloop code [skip ci] * Automated Commit - Formatting Changes * [ISSUE #26581] move partition serialization to JSON * Revert "[ISSUE #26607] zenloop migration (#27243)" This reverts commit 5c6f19b. * [ISSUE #26607] revert zenloop --------- Co-authored-by: Augustin <augustin@airbyte.io> Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>

* [ISSUE #26581] per partition cursor * [ISSUE #26581] format * [ISSUE #26581] clean up state management * [ISSUE #26581] improving Hashabledict * [ISSUE #26581] format cdk * [ISSUE #26581] fix tests * [ISSUE #26581] code review from girarda * Retrigger pipeline * [ISSUE #26581] code review * Automated Commit - Formatting Changes * [ISSUE #26581] validation overlapping keys * [ISSUE #26581] add typing * [ISSUE #26581] code review * [ISSUE #26607] zenloop migration (#27243) * [ISSUE #26607] zenloop migration implementation without tests * [ISSUE #26607] zenloop migration adding edge cases * [ISSUE #26607] add cursor field for state * [ISSUE #26607] update abnormal state * [ISSUE #26607] ensure default state * [ISSUE #26607] updating CATs state * [ISSUE #26607] revert migrating cursor * [ISSUE #26607] remove default cursor value * [ISSUE #26607] improve error message * [ISSUE #26607] changelog --------- Co-authored-by: Augustin <augustin@airbyte.io> * 🤖 Auto format source-zenloop code [skip ci] * Automated Commit - Formatting Changes * [ISSUE #26581] move partition serialization to JSON * Revert "[ISSUE #26607] zenloop migration (#27243)" This reverts commit 5c6f19b. * Revert "Revert "[ISSUE #26607] zenloop migration (#27243)"" This reverts commit e363fd6. * [ISSUE #26607] update zenloop version * TMP specify cdk version * [ISSUE #26607] do not lock zenloop airbyte_cdk version * trigger pipeline * Automated Commit - Formatting Changes * trigger pipeline --------- Co-authored-by: Augustin <augustin@airbyte.io> Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>

[ISSUE #26607] zenloop migration implementation without tests

64f8f65

octavia-squidington-iii added the CDK Connector Development Kit label Jun 12, 2023

[ISSUE #26607] zenloop migration adding edge cases

569cddf

[ISSUE #26607] add cursor field for state

0e94475

octavia-squidington-iii added area/connectors Connector related issues connectors/source/zenloop labels Jun 12, 2023

maxi297 added 4 commits June 12, 2023 16:18

[ISSUE #26607] update abnormal state

672b34d

[ISSUE #26607] ensure default state

725c053

[ISSUE #26607] updating CATs state

8925987

[ISSUE #26607] revert migrating cursor

9378b9a

maxi297 changed the title ~~[ISSUE #26607] zenloop migration implementation without tests~~ [ISSUE #26607] zenloop migration Jun 14, 2023

[ISSUE #26607] remove default cursor value

ca3a1cd

maxi297 added 2 commits June 14, 2023 10:12

Merge branch 'issue-26581/per-partition-cursor' into issue-26607/zenl…

6ce627d

…oop-migration-to-per-partition-cursor

[ISSUE #26607] improve error message

95eaf03

maxi297 marked this pull request as ready for review June 14, 2023 16:35

maxi297 requested a review from a team as a code owner June 14, 2023 16:35

maxi297 requested review from girarda and lazebnyi June 14, 2023 16:35

[ISSUE #26607] changelog

025c4e7

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Jun 14, 2023

Merge branch 'issue-26581/per-partition-cursor' into issue-26607/zenl…

553621a

…oop-migration-to-per-partition-cursor

maxi297 mentioned this pull request Jun 14, 2023

Issue 26581/per partition cursor #27223

Merged

girarda approved these changes Jun 15, 2023

View reviewed changes

lazebnyi approved these changes Jun 16, 2023

View reviewed changes

maxi297 force-pushed the issue-26581/per-partition-cursor branch from 3593d43 to 86c594a Compare June 16, 2023 19:14

Merge branch 'issue-26581/per-partition-cursor' into issue-26607/zenl…

9f18ccb

…oop-migration-to-per-partition-cursor

maxi297 merged commit 5c6f19b into issue-26581/per-partition-cursor Jun 21, 2023
19 of 22 checks passed

maxi297 deleted the issue-26607/zenloop-migration-to-per-partition-cursor branch June 21, 2023 12:41

maxi297 added a commit that referenced this pull request Jun 21, 2023

Revert "[ISSUE #26607] zenloop migration (#27243)"

e363fd6

This reverts commit 5c6f19b.

maxi297 added a commit that referenced this pull request Jun 21, 2023

Revert "Revert "[ISSUE #26607] zenloop migration (#27243)""

82066b8

This reverts commit e363fd6.

maxi297 mentioned this pull request Jun 22, 2023

🐛 Source zenloop: update to state per partition #27556

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ISSUE #26607] zenloop migration #27243

[ISSUE #26607] zenloop migration #27243

maxi297 commented Jun 12, 2023 •

edited

maxi297 commented Jun 12, 2023 •

edited by github-actions bot

maxi297 commented Jun 12, 2023 •

edited by github-actions bot

maxi297 commented Jun 12, 2023 •

edited by github-actions bot

maxi297 commented Jun 14, 2023

maxi297 commented Jun 14, 2023 •

edited by github-actions bot

alafanechere commented Jun 14, 2023

alafanechere commented Jun 14, 2023

maxi297 commented Jun 14, 2023

octavia-squidington-iii commented Jun 16, 2023

[ISSUE #26607] zenloop migration #27243

[ISSUE #26607] zenloop migration #27243

Conversation

maxi297 commented Jun 12, 2023 • edited

What

How

🚨 User Impact 🚨

maxi297 commented Jun 12, 2023 • edited by github-actions bot

Build Failed

maxi297 commented Jun 12, 2023 • edited by github-actions bot

Build Failed

maxi297 commented Jun 12, 2023 • edited by github-actions bot

Build Failed

maxi297 commented Jun 14, 2023

maxi297 commented Jun 14, 2023 • edited by github-actions bot

Build Passed

alafanechere commented Jun 14, 2023

alafanechere commented Jun 14, 2023

maxi297 commented Jun 14, 2023

octavia-squidington-iii commented Jun 16, 2023

source-zenloop test report (commit 9f18ccb2db) - ❌

maxi297 commented Jun 12, 2023 •

edited

maxi297 commented Jun 12, 2023 •

edited by github-actions bot

maxi297 commented Jun 12, 2023 •

edited by github-actions bot

maxi297 commented Jun 12, 2023 •

edited by github-actions bot

maxi297 commented Jun 14, 2023 •

edited by github-actions bot

source-zenloop test report (commit `9f18ccb2db`) - ❌