Publish stream status messages in CDK #24994

jdpgrailsdev · 2023-04-07T18:49:53Z

What

Report transitions in status while reading a source

How

Publishes the stream status message on the various transitions in status for each stream in the source. This is a parallel PR to #24971.

Recommended reading order

abstract_source.py
stream_status_utils.py

jdpgrailsdev · 2023-04-13T20:49:14Z

This PR has been updated to use the new protocol message and is now ready for review.

evantahler · 2023-04-14T16:17:30Z

airbyte-cdk/python/setup.py

-        "airbyte-protocol-models==1.0.0",
+        "airbyte-protocol-models==0.3.5",


This looks like a downgrade... but I don't think we ever were on v1.0.0?

Correct. We were never on 1.0.0 even though we set that as the dependency version. The new changes were released as a patch on the current version (0.3), which is the reason for the correction.

airbyte-cdk/python/airbyte_cdk/sources/abstract_source.py

…ytehq/airbyte into jonathan/cdk-stream-status-messages

girarda · 2023-04-19T21:21:45Z

/test connector=connectors/source-stripe local_cdk=1

🕑 connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/4748046844
❌ connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/4748046844
🐛 https://gradle.com/s/aq2op5ellhazu

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Please check...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
============= 1 failed, 37 passed, 2 skipped in 228.23s (0:03:48) ==============

girarda · 2023-04-19T22:27:10Z

/test connector=connectors/source-greenhouse local_cdk=1

🕑 connectors/source-greenhouse https://github.com/airbytehq/airbyte/actions/runs/4748479147
✅ connectors/source-greenhouse https://github.com/airbytehq/airbyte/actions/runs/4748479147
Python tests coverage:

Name                              Stmts   Miss  Cover
-----------------------------------------------------
source_greenhouse/source.py           4      0   100%
source_greenhouse/__init__.py         2      0   100%
source_greenhouse/components.py      68      5    93%
-----------------------------------------------------
TOTAL                                74      5    93%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
================== 47 passed, 3 skipped in 289.49s (0:04:49) ===================

girarda · 2023-04-19T22:46:28Z

/test connector=connectors/source-facebook-marketing local_cdk=1

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/4748592172
❌ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/4748592172
🐛 https://gradle.com/s/yuk4xsok4u4ra

Build Failed

Test summary info:

Could not find result summary

girarda · 2023-04-20T15:53:02Z

/test connector=connectors/source-facebook-marketing local_cdk=1

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/4756325339
❌ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/4756325339
🐛 https://gradle.com/s/yxxemn63g55fy

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream ads_i...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
============= 1 failed, 37 passed, 2 skipped in 1373.80s (0:22:53) =============

girarda · 2023-04-20T16:28:48Z

/test connector=connectors/source-facebook-marketing

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/4756652161
❌ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/4756652161
🐛 https://gradle.com/s/wyftobx7vcytw

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream ads_i...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
============= 2 failed, 36 passed, 2 skipped in 1523.49s (0:25:23) =============

girarda · 2023-04-21T16:52:36Z

/test connector=connectors/source-greenhouse local_cdk=1

🕑 connectors/source-greenhouse https://github.com/airbytehq/airbyte/actions/runs/4767071665
✅ connectors/source-greenhouse https://github.com/airbytehq/airbyte/actions/runs/4767071665
Python tests coverage:

Name                              Stmts   Miss  Cover
-----------------------------------------------------
source_greenhouse/source.py           4      0   100%
source_greenhouse/__init__.py         2      0   100%
source_greenhouse/components.py      68      5    93%
-----------------------------------------------------
TOTAL                                74      5    93%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
================== 47 passed, 3 skipped in 323.59s (0:05:23) ===================

girarda

jdpgrailsdev · 2023-04-25T22:22:10Z

/test connector=connectors/source-facebook-marketing local_cdk=1

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/4802745891
✅ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/4802745891
Python tests coverage:

Name                                                        Stmts   Miss  Cover
-------------------------------------------------------------------------------
source_facebook_marketing/utils.py                             26      0   100%
source_facebook_marketing/streams/common.py                    44      0   100%
source_facebook_marketing/streams/async_job.py                217      0   100%
source_facebook_marketing/streams/__init__.py                   2      0   100%
source_facebook_marketing/spec.py                              42      0   100%
source_facebook_marketing/__init__.py                           5      0   100%
source_facebook_marketing/streams/async_job_manager.py         78      3    96%
source_facebook_marketing/streams/patches.py                   22      2    91%
source_facebook_marketing/api.py                               96      9    91%
source_facebook_marketing/source.py                            61      6    90%
source_facebook_marketing/streams/base_insight_streams.py     145     16    89%
source_facebook_marketing/streams/base_streams.py             143     22    85%
source_facebook_marketing/streams/streams.py                  125     29    77%
-------------------------------------------------------------------------------
TOTAL                                                        1006     87    91%
Name                                                        Stmts   Miss  Cover
-------------------------------------------------------------------------------
source_facebook_marketing/streams/__init__.py                   2      0   100%
source_facebook_marketing/__init__.py                           5      0   100%
source_facebook_marketing/spec.py                              42      1    98%
source_facebook_marketing/api.py                               96     12    88%
source_facebook_marketing/streams/patches.py                   22      3    86%
source_facebook_marketing/streams/base_streams.py             143     28    80%
source_facebook_marketing/streams/common.py                    44     16    64%
source_facebook_marketing/utils.py                             26     10    62%
source_facebook_marketing/source.py                            61     24    61%
source_facebook_marketing/streams/streams.py                  125     50    60%
source_facebook_marketing/streams/base_insight_streams.py     145     64    56%
source_facebook_marketing/streams/async_job.py                217    136    37%
source_facebook_marketing/streams/async_job_manager.py         78     60    23%
-------------------------------------------------------------------------------
TOTAL                                                        1006    404    60%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
================== 38 passed, 2 skipped in 1506.80s (0:25:06) ==================

This reverts commit 709edb8.

jdpgrailsdev · 2023-04-26T13:29:37Z

Per a discussion with @alafanechere I have reverted the changelog/version bump to the source-facebook-marketing connector to avoid an order of operation issue between updating and publishing the CDK and publishing the connector. This PR will be merged first and once the CDK has been published, a second PR will be created to publish the source-facebook-marketing connector.

* Publish stream status messages in CDK * Automated Commit - Formatting Changes * Convert to StreamDescriptor * Automated Commit - Formatting Changes * Bump to latest protocol model * Automated Commit - Formatting Changes * Bump protocol version * Add tests for stream status message creation * Formatting * Formatting * Fix failing test * Actually emit state message * Automated Commit - Formatting Changes * Bump airbyte-protocol * PR feedback * Fix parameter input * Correctly yield status message * PR feedback * Formatting * Fix failing tests * Automated Commit - Formatting Changes * Revert accidental change * Automated Change * Replace STOPPED with COMPLETE/INCOMPLETE * Update source-facebook-marketing changelog * Revert "Update source-facebook-marketing changelog" This reverts commit 709edb8. --------- Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>

jdpgrailsdev requested a review from davinchia April 7, 2023 18:49

octavia-squidington-iii added the CDK Connector Development Kit label Apr 7, 2023

jdpgrailsdev and others added 5 commits April 13, 2023 10:43

Publish stream status messages in CDK

3ed82e1

Automated Commit - Formatting Changes

d49e33d

Convert to StreamDescriptor

52b7c66

Automated Commit - Formatting Changes

361b7b5

Bump to latest protocol model

b716aba

jdpgrailsdev force-pushed the jonathan/cdk-stream-status-messages branch from 12a5754 to b716aba Compare April 13, 2023 14:43

jdpgrailsdev and others added 10 commits April 13, 2023 14:47

Automated Commit - Formatting Changes

681cc58

Merge branch 'master' into jonathan/cdk-stream-status-messages

31cbde1

Bump protocol version

c31f0bc

Add tests for stream status message creation

6735aee

Formatting

f53ef81

Formatting

105a1a2

Fix failing test

f64867f

Actually emit state message

89b4e50

Automated Commit - Formatting Changes

5ba0e30

Merge branch 'master' into jonathan/cdk-stream-status-messages

8752a28

jdpgrailsdev changed the title ~~WIP Publish stream status messages in CDK~~ Publish stream status messages in CDK Apr 13, 2023

Merge branch 'master' into jonathan/cdk-stream-status-messages

b242ec4

jdpgrailsdev marked this pull request as ready for review April 13, 2023 20:49

jdpgrailsdev requested a review from a team as a code owner April 13, 2023 20:49

jdpgrailsdev requested review from girarda and alafanechere April 13, 2023 20:49

jdpgrailsdev and others added 2 commits April 14, 2023 09:05

Bump airbyte-protocol

21f4e8e

Merge branch 'master' into jonathan/cdk-stream-status-messages

9e063b8

evantahler reviewed Apr 14, 2023

View reviewed changes

Merge branch 'master' into jonathan/cdk-stream-status-messages

8f02fce

girarda reviewed Apr 14, 2023

View reviewed changes

airbyte-cdk/python/airbyte_cdk/sources/abstract_source.py Outdated Show resolved Hide resolved

Merge branch 'jonathan/cdk-stream-status-messages' of github.com:airb…

518a34d

…ytehq/airbyte into jonathan/cdk-stream-status-messages

jdpgrailsdev and others added 2 commits April 20, 2023 10:14

Merge branch 'master' into jonathan/cdk-stream-status-messages

a2de336

Automated Change

3ae7137

jdpgrailsdev and others added 2 commits April 21, 2023 09:17

Merge branch 'master' into jonathan/cdk-stream-status-messages

4ae440b

Replace STOPPED with COMPLETE/INCOMPLETE

718453e

girarda approved these changes Apr 21, 2023

View reviewed changes

jdpgrailsdev and others added 5 commits April 21, 2023 15:27

Merge branch 'master' into jonathan/cdk-stream-status-messages

c381cec

Merge branch 'master' into jonathan/cdk-stream-status-messages

90d6202

Merge branch 'master' into jonathan/cdk-stream-status-messages

d6e676e

Merge branch 'master' into jonathan/cdk-stream-status-messages

e80397e

Update source-facebook-marketing changelog

709edb8

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/facebook-marketing labels Apr 25, 2023

Revert "Update source-facebook-marketing changelog"

95cffdb

This reverts commit 709edb8.

jdpgrailsdev force-pushed the jonathan/cdk-stream-status-messages branch from d8321ee to 95cffdb Compare April 26, 2023 13:09

octavia-squidington-iii removed area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Apr 26, 2023

Merge branch 'master' into jonathan/cdk-stream-status-messages

569cce5

jdpgrailsdev merged commit 2ebfa45 into master Apr 26, 2023

jdpgrailsdev deleted the jonathan/cdk-stream-status-messages branch April 26, 2023 14:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Publish stream status messages in CDK #24994

Publish stream status messages in CDK #24994

jdpgrailsdev commented Apr 7, 2023 •

edited

Loading

jdpgrailsdev commented Apr 13, 2023

evantahler Apr 14, 2023

jdpgrailsdev Apr 15, 2023

girarda commented Apr 19, 2023 •

edited by github-actions bot

Loading

girarda commented Apr 19, 2023 •

edited by github-actions bot

Loading

girarda commented Apr 19, 2023 •

edited by github-actions bot

Loading

girarda commented Apr 20, 2023 •

edited by github-actions bot

Loading

girarda commented Apr 20, 2023 •

edited by github-actions bot

Loading

girarda commented Apr 21, 2023 •

edited by github-actions bot

Loading

girarda left a comment

jdpgrailsdev commented Apr 25, 2023 •

edited by github-actions bot

Loading

jdpgrailsdev commented Apr 26, 2023

		"airbyte-protocol-models==1.0.0",
		"airbyte-protocol-models==0.3.5",

Publish stream status messages in CDK #24994

Publish stream status messages in CDK #24994

Conversation

jdpgrailsdev commented Apr 7, 2023 • edited Loading

What

How

Recommended reading order

jdpgrailsdev commented Apr 13, 2023

evantahler Apr 14, 2023

Choose a reason for hiding this comment

jdpgrailsdev Apr 15, 2023

Choose a reason for hiding this comment

girarda commented Apr 19, 2023 • edited by github-actions bot Loading

Build Failed

girarda commented Apr 19, 2023 • edited by github-actions bot Loading

Build Passed

girarda commented Apr 19, 2023 • edited by github-actions bot Loading

Build Failed

girarda commented Apr 20, 2023 • edited by github-actions bot Loading

Build Failed

girarda commented Apr 20, 2023 • edited by github-actions bot Loading

Build Failed

girarda commented Apr 21, 2023 • edited by github-actions bot Loading

Build Passed

girarda left a comment

Choose a reason for hiding this comment

jdpgrailsdev commented Apr 25, 2023 • edited by github-actions bot Loading

Build Passed

jdpgrailsdev commented Apr 26, 2023

jdpgrailsdev commented Apr 7, 2023 •

edited

Loading

girarda commented Apr 19, 2023 •

edited by github-actions bot

Loading

girarda commented Apr 19, 2023 •

edited by github-actions bot

Loading

girarda commented Apr 19, 2023 •

edited by github-actions bot

Loading

girarda commented Apr 20, 2023 •

edited by github-actions bot

Loading

girarda commented Apr 20, 2023 •

edited by github-actions bot

Loading

girarda commented Apr 21, 2023 •

edited by github-actions bot

Loading

jdpgrailsdev commented Apr 25, 2023 •

edited by github-actions bot

Loading