[source-sendgrid] [source-sentry] use latest cdk w/ file parse fixes and fix sendgrid messages stream bug #15911

brianjlai · 2022-08-23T23:56:11Z

What

Upgrade the sendgrid and sentry connectors to use the latest CDK which fixes a json schema parse issue as per 0.1.79.

How

The latest CDK can properly read json files via #15814

brianjlai · 2022-08-23T23:56:56Z

airbyte-integrations/connectors/source-sendgrid/source_sendgrid/sendgrid.yaml

@@ -67,7 +67,7 @@ definitions:
      datetime: "{{ config['start_time'] }}"
      datetime_format: "%s"
    end_datetime:
-      datetime: "{{ now_utc() }}}"


fixes time parsing that was failing when it got an unexpected }

Did this cause any test failures?

It didn't, but i think the reason was because the messages stream uses an API endpoint that our test creds don't have access to which then returns a 400 error code. I haven't dug into how our unit tests from a month or two ago are working, but its possible they see the same error and move on to the next set of streams

brianjlai · 2022-08-24T05:43:39Z

/test connector=connectors/source-sentry

🕑 connectors/source-sentry https://github.com/airbytehq/airbyte/actions/runs/2916728281
✅ connectors/source-sentry https://github.com/airbytehq/airbyte/actions/runs/2916728281
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1322    463    65%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
======================== 26 passed, 1 skipped in 33.21s ========================

brianjlai · 2022-08-24T05:43:52Z

/test connector=connectors/source-sendgrid

🕑 connectors/source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/2916729382
✅ connectors/source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/2916729382
Python tests coverage:

Name                          Stmts   Miss  Cover
-------------------------------------------------
source_sendgrid/source.py         4      0   100%
source_sendgrid/__init__.py       2      0   100%
-------------------------------------------------
TOTAL                             6      0   100%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1322    463    65%

Build Passed

Test summary info:

All Passed

pedroslopez

LGTM (remember to update changelog)

alafanechere

LGTM, missing changelog update and connector version bump.

brianjlai · 2022-08-24T17:29:46Z

/publish connector=connectors/source-sendgrid

🕑 Publishing the following connectors:
connectors/source-sendgrid
https://github.com/airbytehq/airbyte/actions/runs/2921092169

Connector	Did it publish?	Were definitions generated?
connectors/source-sendgrid	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

brianjlai · 2022-08-24T17:29:52Z

/publish connector=connectors/source-sentry

🕑 Publishing the following connectors:
connectors/source-sentry
https://github.com/airbytehq/airbyte/actions/runs/2921092912

Connector	Did it publish?	Were definitions generated?
connectors/source-sentry	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…thub.com/airbytehq/airbyte into brian/bump_sendgrid_sentry_to_latest_CDK

…and fix sendgrid messages stream bug (#15911) * use latest cdk w/ file parse fixes and fix sendgrid messages stream bug * roll dockerfile and changelog * auto-bump connector version [ci skip] * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

use latest cdk w/ file parse fixes and fix sendgrid messages stream bug

ee680b8

github-actions bot added the area/connectors Connector related issues label Aug 23, 2022

octavia-squidington-iii added connectors/source/sentry connectors/source/sendgrid labels Aug 23, 2022

brianjlai changed the title ~~use latest cdk w/ file parse fixes and fix sendgrid messages stream bug~~ [source-sendgrid] [source-sentry] use latest cdk w/ file parse fixes and fix sendgrid messages stream bug Aug 23, 2022

brianjlai commented Aug 23, 2022

View reviewed changes

brianjlai requested review from sherifnada, pedroslopez, alafanechere and Phlair August 23, 2022 23:57

pedroslopez approved these changes Aug 24, 2022

View reviewed changes

alafanechere approved these changes Aug 24, 2022

View reviewed changes

roll dockerfile and changelog

189ee60

github-actions bot added the area/documentation Improvements or additions to documentation label Aug 24, 2022

octavia-squidington-iii added 3 commits August 24, 2022 17:51

auto-bump connector version [ci skip]

df5905c

auto-bump connector version [ci skip]

37fd8de

Merge branch 'brian/bump_sendgrid_sentry_to_latest_CDK' of https://gi…

4d47d95

…thub.com/airbytehq/airbyte into brian/bump_sendgrid_sentry_to_latest_CDK

octavia-squidington-iii temporarily deployed to more-secrets August 24, 2022 18:22 Inactive

brianjlai merged commit 63d0fa1 into master Aug 24, 2022

brianjlai deleted the brian/bump_sendgrid_sentry_to_latest_CDK branch August 24, 2022 18:58

octavia-squidington-iii mentioned this pull request Aug 26, 2022

Bump Airbyte version from 0.40.2 to 0.40.3 #16028

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[source-sendgrid] [source-sentry] use latest cdk w/ file parse fixes and fix sendgrid messages stream bug #15911

[source-sendgrid] [source-sentry] use latest cdk w/ file parse fixes and fix sendgrid messages stream bug #15911

brianjlai commented Aug 23, 2022 •

edited

Loading

brianjlai Aug 23, 2022 •

edited

Loading

pedroslopez Aug 24, 2022

brianjlai Aug 24, 2022

brianjlai commented Aug 24, 2022 •

edited by github-actions bot

Loading

brianjlai commented Aug 24, 2022 •

edited by github-actions bot

Loading

pedroslopez left a comment

alafanechere left a comment

brianjlai commented Aug 24, 2022 •

edited by github-actions bot

Loading

brianjlai commented Aug 24, 2022 •

edited by github-actions bot

Loading

[source-sendgrid] [source-sentry] use latest cdk w/ file parse fixes and fix sendgrid messages stream bug #15911

[source-sendgrid] [source-sentry] use latest cdk w/ file parse fixes and fix sendgrid messages stream bug #15911

Conversation

brianjlai commented Aug 23, 2022 • edited Loading

What

How

brianjlai Aug 23, 2022 • edited Loading

Choose a reason for hiding this comment

pedroslopez Aug 24, 2022

Choose a reason for hiding this comment

brianjlai Aug 24, 2022

Choose a reason for hiding this comment

brianjlai commented Aug 24, 2022 • edited by github-actions bot Loading

Build Passed

brianjlai commented Aug 24, 2022 • edited by github-actions bot Loading

Build Passed

pedroslopez left a comment

Choose a reason for hiding this comment

alafanechere left a comment

Choose a reason for hiding this comment

brianjlai commented Aug 24, 2022 • edited by github-actions bot Loading

brianjlai commented Aug 24, 2022 • edited by github-actions bot Loading

brianjlai commented Aug 23, 2022 •

edited

Loading

brianjlai Aug 23, 2022 •

edited

Loading

brianjlai commented Aug 24, 2022 •

edited by github-actions bot

Loading

brianjlai commented Aug 24, 2022 •

edited by github-actions bot

Loading

brianjlai commented Aug 24, 2022 •

edited by github-actions bot

Loading

brianjlai commented Aug 24, 2022 •

edited by github-actions bot

Loading