Publish stream status messages in CDK by jdpgrailsdev · Pull Request #24994 · airbytehq/airbyte

Jonathan Pearlin (jdpgrailsdev) · 2023-04-07T18:49:53Z

What

Report transitions in status while reading a source

How

Publishes the stream status message on the various transitions in status for each stream in the source. This is a parallel PR to #24971.

Recommended reading order

abstract_source.py
stream_status_utils.py

Jonathan Pearlin (jdpgrailsdev) · 2023-04-13T20:49:14Z

This PR has been updated to use the new protocol message and is now ready for review.

Evan Tahler (evantahler) · 2023-04-14T16:17:30Z

    package_data={"airbyte_cdk": ["py.typed", "sources/declarative/declarative_component_schema.yaml"]},
    install_requires=[
-        "airbyte-protocol-models==1.0.0",
+        "airbyte-protocol-models==0.3.5",


This looks like a downgrade... but I don't think we ever were on v1.0.0?

Correct. We were never on 1.0.0 even though we set that as the dependency version. The new changes were released as a patch on the current version (0.3), which is the reason for the correction.

…ytehq/airbyte into jonathan/cdk-stream-status-messages

Alexandre Girard (girarda) · 2023-04-19T21:21:45Z

/test connector=connectors/source-stripe local_cdk=1

🕑 connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/4748046844
❌ connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/4748046844
🐛 https://gradle.com/s/aq2op5ellhazu

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Please check...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
============= 1 failed, 37 passed, 2 skipped in 228.23s (0:03:48) ==============

Alexandre Girard (girarda) · 2023-04-19T22:27:10Z

/test connector=connectors/source-greenhouse local_cdk=1

🕑 connectors/source-greenhouse https://github.com/airbytehq/airbyte/actions/runs/4748479147
✅ connectors/source-greenhouse https://github.com/airbytehq/airbyte/actions/runs/4748479147
Python tests coverage:

Name                              Stmts   Miss  Cover
-----------------------------------------------------
source_greenhouse/source.py           4      0   100%
source_greenhouse/__init__.py         2      0   100%
source_greenhouse/components.py      68      5    93%
-----------------------------------------------------
TOTAL                                74      5    93%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
================== 47 passed, 3 skipped in 289.49s (0:04:49) ===================

Alexandre Girard (girarda) · 2023-04-19T22:46:28Z

/test connector=connectors/source-facebook-marketing local_cdk=1

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/4748592172
❌ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/4748592172
🐛 https://gradle.com/s/yuk4xsok4u4ra

Build Failed

Test summary info:

Could not find result summary

Alexandre Girard (girarda) · 2023-04-20T15:53:02Z

/test connector=connectors/source-facebook-marketing local_cdk=1

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/4756325339
❌ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/4756325339
🐛 https://gradle.com/s/yxxemn63g55fy

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream ads_i...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
============= 1 failed, 37 passed, 2 skipped in 1373.80s (0:22:53) =============

Alexandre Girard (girarda) · 2023-04-20T16:28:48Z

/test connector=connectors/source-facebook-marketing

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/4756652161
❌ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/4756652161
🐛 https://gradle.com/s/wyftobx7vcytw

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream ads_i...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
============= 2 failed, 36 passed, 2 skipped in 1523.49s (0:25:23) =============

Alexandre Girard (girarda) · 2023-04-21T16:52:36Z

/test connector=connectors/source-greenhouse local_cdk=1

🕑 connectors/source-greenhouse https://github.com/airbytehq/airbyte/actions/runs/4767071665
✅ connectors/source-greenhouse https://github.com/airbytehq/airbyte/actions/runs/4767071665
Python tests coverage:

Name                              Stmts   Miss  Cover
-----------------------------------------------------
source_greenhouse/source.py           4      0   100%
source_greenhouse/__init__.py         2      0   100%
source_greenhouse/components.py      68      5    93%
-----------------------------------------------------
TOTAL                                74      5    93%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
================== 47 passed, 3 skipped in 323.59s (0:05:23) ===================

Alexandre Girard (girarda)

Jonathan Pearlin (jdpgrailsdev) · 2023-04-25T22:22:10Z

/test connector=connectors/source-facebook-marketing local_cdk=1

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/4802745891
✅ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/4802745891
Python tests coverage:

Name                                                        Stmts   Miss  Cover
-------------------------------------------------------------------------------
source_facebook_marketing/utils.py                             26      0   100%
source_facebook_marketing/streams/common.py                    44      0   100%
source_facebook_marketing/streams/async_job.py                217      0   100%
source_facebook_marketing/streams/__init__.py                   2      0   100%
source_facebook_marketing/spec.py                              42      0   100%
source_facebook_marketing/__init__.py                           5      0   100%
source_facebook_marketing/streams/async_job_manager.py         78      3    96%
source_facebook_marketing/streams/patches.py                   22      2    91%
source_facebook_marketing/api.py                               96      9    91%
source_facebook_marketing/source.py                            61      6    90%
source_facebook_marketing/streams/base_insight_streams.py     145     16    89%
source_facebook_marketing/streams/base_streams.py             143     22    85%
source_facebook_marketing/streams/streams.py                  125     29    77%
-------------------------------------------------------------------------------
TOTAL                                                        1006     87    91%
Name                                                        Stmts   Miss  Cover
-------------------------------------------------------------------------------
source_facebook_marketing/streams/__init__.py                   2      0   100%
source_facebook_marketing/__init__.py                           5      0   100%
source_facebook_marketing/spec.py                              42      1    98%
source_facebook_marketing/api.py                               96     12    88%
source_facebook_marketing/streams/patches.py                   22      3    86%
source_facebook_marketing/streams/base_streams.py             143     28    80%
source_facebook_marketing/streams/common.py                    44     16    64%
source_facebook_marketing/utils.py                             26     10    62%
source_facebook_marketing/source.py                            61     24    61%
source_facebook_marketing/streams/streams.py                  125     50    60%
source_facebook_marketing/streams/base_insight_streams.py     145     64    56%
source_facebook_marketing/streams/async_job.py                217    136    37%
source_facebook_marketing/streams/async_job_manager.py         78     60    23%
-------------------------------------------------------------------------------
TOTAL                                                        1006    404    60%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
================== 38 passed, 2 skipped in 1506.80s (0:25:06) ==================

This reverts commit 709edb8.

Jonathan Pearlin (jdpgrailsdev) · 2023-04-26T13:29:37Z

Per a discussion with Augustin (@alafanechere) I have reverted the changelog/version bump to the source-facebook-marketing connector to avoid an order of operation issue between updating and publishing the CDK and publishing the connector. This PR will be merged first and once the CDK has been published, a second PR will be created to publish the source-facebook-marketing connector.

* Publish stream status messages in CDK * Automated Commit - Formatting Changes * Convert to StreamDescriptor * Automated Commit - Formatting Changes * Bump to latest protocol model * Automated Commit - Formatting Changes * Bump protocol version * Add tests for stream status message creation * Formatting * Formatting * Fix failing test * Actually emit state message * Automated Commit - Formatting Changes * Bump airbyte-protocol * PR feedback * Fix parameter input * Correctly yield status message * PR feedback * Formatting * Fix failing tests * Automated Commit - Formatting Changes * Revert accidental change * Automated Change * Replace STOPPED with COMPLETE/INCOMPLETE * Update source-facebook-marketing changelog * Revert "Update source-facebook-marketing changelog" This reverts commit 709edb8. --------- Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>

Jonathan Pearlin (jdpgrailsdev) requested a review from Davin Chia (davinchia) April 7, 2023 18:49

Octavia Squidington III (octavia-squidington-iii) added the CDK Connector Development Kit label Apr 7, 2023

Jonathan Pearlin (jdpgrailsdev) and others added 5 commits April 13, 2023 10:43

Publish stream status messages in CDK

3ed82e1

Automated Commit - Formatting Changes

d49e33d

Convert to StreamDescriptor

52b7c66

Automated Commit - Formatting Changes

361b7b5

Bump to latest protocol model

b716aba

Jonathan Pearlin (jdpgrailsdev) force-pushed the jonathan/cdk-stream-status-messages branch from 12a5754 to b716aba Compare April 13, 2023 14:43

Jonathan Pearlin (jdpgrailsdev) and others added 10 commits April 13, 2023 14:47

Automated Commit - Formatting Changes

681cc58

Merge branch 'master' into jonathan/cdk-stream-status-messages

31cbde1

Bump protocol version

c31f0bc

Add tests for stream status message creation

6735aee

Formatting

f53ef81

Formatting

105a1a2

Fix failing test

f64867f

Actually emit state message

89b4e50

Automated Commit - Formatting Changes

5ba0e30

Merge branch 'master' into jonathan/cdk-stream-status-messages

8752a28

Jonathan Pearlin (jdpgrailsdev) changed the title ~~WIP Publish stream status messages in CDK~~ Publish stream status messages in CDK Apr 13, 2023

Merge branch 'master' into jonathan/cdk-stream-status-messages

b242ec4

Jonathan Pearlin (jdpgrailsdev) marked this pull request as ready for review April 13, 2023 20:49

Jonathan Pearlin (jdpgrailsdev) requested review from a team, Augustin (alafanechere) and Alexandre Girard (girarda) April 13, 2023 20:49

Jonathan Pearlin (jdpgrailsdev) and others added 2 commits April 14, 2023 09:05

Bump airbyte-protocol

21f4e8e

Merge branch 'master' into jonathan/cdk-stream-status-messages

9e063b8

Evan Tahler (evantahler) reviewed Apr 14, 2023

View reviewed changes

Merge branch 'master' into jonathan/cdk-stream-status-messages

8f02fce

Alexandre Girard (girarda) reviewed Apr 14, 2023

View reviewed changes

Comment thread airbyte-cdk/python/airbyte_cdk/sources/abstract_source.py Outdated

Merge branch 'jonathan/cdk-stream-status-messages' of github.com:airb…

518a34d

…ytehq/airbyte into jonathan/cdk-stream-status-messages

Jonathan Pearlin and others added 2 commits April 20, 2023 10:14

Merge branch 'master' into jonathan/cdk-stream-status-messages

a2de336

Automated Change

3ae7137

Jonathan Pearlin and others added 2 commits April 21, 2023 09:17

Merge branch 'master' into jonathan/cdk-stream-status-messages

4ae440b

Replace STOPPED with COMPLETE/INCOMPLETE

718453e

Alexandre Girard (girarda) approved these changes Apr 21, 2023

View reviewed changes

Jonathan Pearlin and others added 5 commits April 21, 2023 15:27

Merge branch 'master' into jonathan/cdk-stream-status-messages

c381cec

Merge branch 'master' into jonathan/cdk-stream-status-messages

90d6202

Merge branch 'master' into jonathan/cdk-stream-status-messages

d6e676e

Merge branch 'master' into jonathan/cdk-stream-status-messages

e80397e

Update source-facebook-marketing changelog

709edb8

Octavia Squidington III (octavia-squidington-iii) added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/facebook-marketing labels Apr 25, 2023

Revert "Update source-facebook-marketing changelog"

95cffdb

This reverts commit 709edb8.

Jonathan Pearlin (jdpgrailsdev) force-pushed the jonathan/cdk-stream-status-messages branch from d8321ee to 95cffdb Compare April 26, 2023 13:09

Octavia Squidington III (octavia-squidington-iii) removed area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Apr 26, 2023

Merge branch 'master' into jonathan/cdk-stream-status-messages

569cce5

Jonathan Pearlin (jdpgrailsdev) merged commit 2ebfa45 into master Apr 26, 2023

Jonathan Pearlin (jdpgrailsdev) deleted the jonathan/cdk-stream-status-messages branch April 26, 2023 14:30

Navigation Menu

Repository navigation

Publish stream status messages in CDK - #24994

Publish stream status messages in CDK#24994
Jonathan Pearlin (jdpgrailsdev) merged 42 commits into
masterfrom
jonathan/cdk-stream-status-messages

Jonathan Pearlin (jdpgrailsdev) commented Apr 7, 2023 •

edited

Loading

Uh oh!

Jonathan Pearlin (jdpgrailsdev) commented Apr 13, 2023

Uh oh!

Evan Tahler (evantahler) Apr 14, 2023

Uh oh!

Jonathan Pearlin (jdpgrailsdev) Apr 15, 2023

Uh oh!

Uh oh!

Alexandre Girard (girarda) commented Apr 19, 2023 •

edited by github-actions Bot

Loading

Uh oh!

Alexandre Girard (girarda) commented Apr 19, 2023 •

edited by github-actions Bot

Loading

Uh oh!

Alexandre Girard (girarda) commented Apr 19, 2023 •

edited by github-actions Bot

Loading

Uh oh!

Alexandre Girard (girarda) commented Apr 20, 2023 •

edited by github-actions Bot

Loading

Uh oh!

Alexandre Girard (girarda) commented Apr 20, 2023 •

edited by github-actions Bot

Loading

Uh oh!

Alexandre Girard (girarda) commented Apr 21, 2023 •

edited by github-actions Bot

Loading

Uh oh!

Alexandre Girard (girarda) left a comment

Uh oh!

Jonathan Pearlin (jdpgrailsdev) commented Apr 25, 2023 •

edited by github-actions Bot

Loading

Uh oh!

Jonathan Pearlin (jdpgrailsdev) commented Apr 26, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Jonathan Pearlin (jdpgrailsdev) commented Apr 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Recommended reading order

Uh oh!

Jonathan Pearlin (jdpgrailsdev) commented Apr 13, 2023

Uh oh!

Evan Tahler (evantahler) Apr 14, 2023

Choose a reason for hiding this comment

Uh oh!

Jonathan Pearlin (jdpgrailsdev) Apr 15, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Alexandre Girard (girarda) commented Apr 19, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Failed

Uh oh!

Alexandre Girard (girarda) commented Apr 19, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

Alexandre Girard (girarda) commented Apr 19, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Failed

Uh oh!

Alexandre Girard (girarda) commented Apr 20, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Failed

Uh oh!

Alexandre Girard (girarda) commented Apr 20, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Failed

Uh oh!

Alexandre Girard (girarda) commented Apr 21, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

Alexandre Girard (girarda) left a comment

Choose a reason for hiding this comment

Uh oh!

Jonathan Pearlin (jdpgrailsdev) commented Apr 25, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

Jonathan Pearlin (jdpgrailsdev) commented Apr 26, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Jonathan Pearlin (jdpgrailsdev) commented Apr 7, 2023 •

edited

Loading

Alexandre Girard (girarda) commented Apr 19, 2023 •

edited by github-actions Bot

Loading

Alexandre Girard (girarda) commented Apr 19, 2023 •

edited by github-actions Bot

Loading

Alexandre Girard (girarda) commented Apr 19, 2023 •

edited by github-actions Bot

Loading

Alexandre Girard (girarda) commented Apr 20, 2023 •

edited by github-actions Bot

Loading

Alexandre Girard (girarda) commented Apr 20, 2023 •

edited by github-actions Bot

Loading

Alexandre Girard (girarda) commented Apr 21, 2023 •

edited by github-actions Bot

Loading

Jonathan Pearlin (jdpgrailsdev) commented Apr 25, 2023 •

edited by github-actions Bot

Loading