Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate Sentry for performance and errors tracking. #8248

Merged
merged 10 commits into from Dec 21, 2021

Conversation

avida
Copy link
Contributor

@avida avida commented Nov 25, 2021

Draft PR for #7959

This is a prototype PR of integrating Sentry into connectors for automatic exception reporting and performance monitoring.

It have been tested on couple of connectors by running locally with DSN passed over SENTRY_DSN environment variable.

Here is a link to browse issues created by Sentry https://sentry.io/organizations/zazmic-wl
@sherifnada You should have received an invite to access this link

@github-actions github-actions bot added the CDK Connector Development Kit label Nov 25, 2021
@sherifnada
Copy link
Contributor

@avida thanks for sharing. I like how simple this implementation is. I took a look at sentry -- my guess is that performance monitoring might be the most useful thing for us, since logs can typically tell us what the exceptions were. WDYT?

Also, do you have an idea of what pricing might look like for our cases?

@avida avida temporarily deployed to more-secrets December 9, 2021 14:28 Inactive
@avida avida temporarily deployed to more-secrets December 9, 2021 14:33 Inactive
@avida
Copy link
Contributor Author

avida commented Dec 10, 2021

To enable sentry for any of connector that using cdk you need:

  1. Install proper version of cdk.
  2. set sentry DSN over "SENTRY_DSN" environment variable

@tuliren
Copy link
Contributor

tuliren commented Dec 13, 2021

@avida, could you add to that Sentry project as well? My username is liren@airbyte.io on Sentry. Thank you.

Copy link
Contributor

@sherifnada sherifnada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sergei-solonitcyn can you manage taking this to the finish line? i won't have bandwidth to review

# Conflicts:
#	airbyte-cdk/python/airbyte_cdk/sources/utils/schema_helpers.py
@augan-rymkhan augan-rymkhan temporarily deployed to more-secrets December 17, 2021 12:49 Inactive
@augan-rymkhan augan-rymkhan temporarily deployed to more-secrets December 17, 2021 12:52 Inactive
Add `SENTRY_DSN` environment variable
@sergei-solonitcyn sergei-solonitcyn temporarily deployed to more-secrets December 21, 2021 01:35 Inactive
Add `sentry_sdk` to install requirements
@sergei-solonitcyn sergei-solonitcyn temporarily deployed to more-secrets December 21, 2021 01:47 Inactive
@sergei-solonitcyn sergei-solonitcyn marked this pull request as ready for review December 21, 2021 01:50
@augan-rymkhan augan-rymkhan temporarily deployed to more-secrets December 21, 2021 05:38 Inactive
@github-actions github-actions bot added the area/connectors Connector related issues label Dec 21, 2021
@augan-rymkhan augan-rymkhan temporarily deployed to more-secrets December 21, 2021 11:41 Inactive
@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Dec 21, 2021
@augan-rymkhan augan-rymkhan temporarily deployed to more-secrets December 21, 2021 11:50 Inactive
@augan-rymkhan
Copy link
Contributor

augan-rymkhan commented Dec 21, 2021

/test connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1606676885
✅ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1606676885
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              242     96    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py       101     65    36%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  972    459    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                            Stmts   Miss  Cover
	 -----------------------------------------------------------------------------------
	 source_google_search_console/__init__.py                            2      0   100%
	 source_google_search_console/service_account_authenticator.py      14      6    57%
	 source_google_search_console/source.py                             37     22    41%
	 source_google_search_console/streams.py                           107     31    71%
	 -----------------------------------------------------------------------------------
	 TOTAL                                                             160     59    63%

@jrhizor jrhizor temporarily deployed to more-secrets December 21, 2021 12:07 Inactive
@augan-rymkhan augan-rymkhan temporarily deployed to more-secrets December 21, 2021 12:20 Inactive
@augan-rymkhan
Copy link
Contributor

augan-rymkhan commented Dec 21, 2021

/publish-cdk dry-run=false

🕑 https://github.com/airbytehq/airbyte/actions/runs/1606818382
https://github.com/airbytehq/airbyte/actions/runs/1606818382

@augan-rymkhan
Copy link
Contributor

augan-rymkhan commented Dec 21, 2021

/publish connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1606846859
✅ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1606846859

@jrhizor jrhizor temporarily deployed to more-secrets December 21, 2021 12:55 Inactive
@augan-rymkhan augan-rymkhan temporarily deployed to more-secrets December 21, 2021 13:07 Inactive
@augan-rymkhan augan-rymkhan merged commit fe954c1 into master Dec 21, 2021
@augan-rymkhan augan-rymkhan deleted the drezchykov/sentry branch December 21, 2021 14:02
@VasylLazebnyk
Copy link

closes #7958

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation CDK Connector Development Kit
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Sentry: Obfuscation and dealing with sensitive data
8 participants