Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bump notion, pipedrive, and pinterest to the latest CDK including per-stream changes #16799

Merged
merged 9 commits into from
Sep 21, 2022
Merged

bump notion, pipedrive, and pinterest to the latest CDK including per-stream changes #16799

merged 9 commits into from
Sep 21, 2022

Conversation

brianjlai
Copy link
Contributor

What

Releases per-stream state for source-notion, source-pinterest, and source-pipedrive connectors

How

Redeploy so that they uptake the latest CDK 0.1.87 which includes support for per-stream state.

Also modifies integration_tests to accurately reflect the new input

@github-actions github-actions bot added the area/connectors Connector related issues label Sep 16, 2022
@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Sep 16, 2022
@alafanechere alafanechere linked an issue Sep 16, 2022 that may be closed by this pull request
[per-stream cdk] Migrate 3 connectors to per-stream state #16060
Closed
@alafanechere
Copy link
Contributor

alafanechere commented Sep 21, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-notion

🕑 connectors/source-notion https://github.com/airbytehq/airbyte/actions/runs/3098966374
✅ connectors/source-notion https://github.com/airbytehq/airbyte/actions/runs/3098966374
Python tests coverage:

Name                        Stmts   Miss  Cover
-----------------------------------------------
source_notion/__init__.py       2      0   100%
source_notion/streams.py      121      5    96%
source_notion/source.py        37      5    86%
source_notion/utils.py          9      4    56%
-----------------------------------------------
TOTAL                         169     14    92%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       152     26    83%   21-23, 29-31, 36-43, 48-61, 239, 250-258
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1358    466    66%

Build Passed

Test summary info:

All Passed

@alafanechere
Copy link
Contributor

alafanechere commented Sep 21, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-pipedrive

🕑 connectors/source-pipedrive https://github.com/airbytehq/airbyte/actions/runs/3099003853
✅ connectors/source-pipedrive https://github.com/airbytehq/airbyte/actions/runs/3099003853
Python tests coverage:

Name                           Stmts   Miss  Cover
--------------------------------------------------
source_pipedrive/__init__.py       2      0   100%
source_pipedrive/source.py        36      2    94%
source_pipedrive/streams.py       77      8    90%
--------------------------------------------------
TOTAL                            115     10    91%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       152     26    83%   21-23, 29-31, 36-43, 48-61, 239, 250-258
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1358    466    66%

Build Passed

Test summary info:

All Passed

@alafanechere
Copy link
Contributor

alafanechere commented Sep 21, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-pinterest

🕑 connectors/source-pinterest https://github.com/airbytehq/airbyte/actions/runs/3099028442
✅ connectors/source-pinterest https://github.com/airbytehq/airbyte/actions/runs/3099028442
Python tests coverage:

Name                           Stmts   Miss  Cover
--------------------------------------------------
source_pinterest/utils.py          4      0   100%
source_pinterest/__init__.py       2      0   100%
source_pinterest/source.py       183     12    93%
--------------------------------------------------
TOTAL                            189     12    94%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       152     26    83%   21-23, 29-31, 36-43, 48-61, 239, 250-258
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1358    466    66%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
=================== 35 passed, 2 skipped in 70.64s (0:01:10) ===================

@alafanechere alafanechere marked this pull request as ready for review September 21, 2022 16:14
@alafanechere alafanechere self-assigned this Sep 21, 2022
@alafanechere
Copy link
Contributor

alafanechere commented Sep 21, 2022 •
edited
Loading

I confirm that I successfully ran two incremental syncs with source-notion on both an old Airbyte version without per-stream, and the newest Airbyte version. State persistence in the DB looked fine in both cases.

@alafanechere
Copy link
Contributor

alafanechere commented Sep 21, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/source-notion,connectors/source-pipedrive,connectors/source-pinterest

🕑 Publishing the following connectors:
connectors/source-notion,connectors/source-pipedrive,connectors/source-pinterest
https://github.com/airbytehq/airbyte/actions/runs/3100506413


Connector Did it publish? Were definitions generated?
connectors/source-notion ✅ ✅
connectors/source-pinterest ✅ ✅
connectors/source-pipedrive ✅ ✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@alafanechere alafanechere merged commit d038624 into master Sep 21, 2022
@alafanechere alafanechere deleted the brian/bump_notion_pinterest_chargebee_to_per_stream branch September 21, 2022 23:17
@alafanechere alafanechere temporarily deployed to more-secrets September 21, 2022 23:18 Inactive
This was referenced Sep 22, 2022
Closed
Merged
robbinhan pushed a commit to robbinhan/airbyte that referenced this pull request Sep 29, 2022
@brianjlai @robbinhan
bump notion, pipedrive, and pinterest to the latest CDK including per…
de9f807 
…-stream changes (airbytehq#16799)
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
@brianjlai @jhammarstedt
bump notion, pipedrive, and pinterest to the latest CDK including per…
9bb5353 
…-stream changes (airbytehq#16799)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alafanechere alafanechere alafanechere approved these changes

Assignees

@alafanechere alafanechere

Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/notion connectors/source/pinterest connectors/source/pipedrive
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[per-stream cdk] Migrate 3 connectors to per-stream state
3 participants
@brianjlai @alafanechere @octavia-squidington-iii