🎉 Source Hubspot: Support OAuth #6465

htrueman · 2021-09-27T16:17:58Z

What

Closes #2449

How

Update spec with oauth creds.
Update acceptance test config with oauth tests.
Add oauth test creds configs.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

Update acceptance test config with oauth tests. Add oauth test creds configs.

…spot-support-oath

CLAassistant · 2021-09-27T17:22:54Z

All committers have signed the CLA.

airbyte-integrations/connectors/source-hubspot/source_hubspot/spec.json

airbyte-integrations/connectors/source-hubspot/source_hubspot/api.py

airbyte-integrations/connectors/source-hubspot/acceptance-test-config.yml

Update source acceptance tests. Update spec.json.

sherifnada · 2021-09-28T15:59:14Z

airbyte-integrations/connectors/source-hubspot/acceptance-test-config.yml

-#        subscription_changes: ["timestamp"]
-#        email_events: ["timestamp"]
+      configured_catalog_path: "sample_files/configured_catalog.json"
+      empty_streams:


why are there so many empty streams? what can we do to populate?

That is on me. I'll update you there after I do a research. So I'll either create an issue to fill those streams with data or fill as much streams as possible.

There is an issue #6711

airbyte-integrations/connectors/source-hubspot/acceptance-test-config.yml

airbyte-integrations/connectors/source-hubspot/source_hubspot/api.py

airbyte-integrations/connectors/source-hubspot/source_hubspot/spec.json

airbyte-integrations/connectors/source-hubspot/source_hubspot/api.py

…ired-scopes-doc' into htrueman/source-hubspot-support-oath

Update comments.

htrueman · 2021-09-29T15:35:13Z

/test connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/1287500529
✅ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/1287500529
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              158    109    31%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     25    39%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py      75     11    85%
	 ------------------------------------------------------------------------
	 TOTAL                                                  778    432    44%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            83     59    29%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     19    42%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        660    393    40%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                         Stmts   Miss  Cover
	 ------------------------------------------------
	 main_dev.py                      6      6     0%
	 source_hubspot/__init__.py       2      0   100%
	 source_hubspot/api.py          364    131    64%
	 source_hubspot/client.py        37     12    68%
	 source_hubspot/errors.py         6      0   100%
	 source_hubspot/source.py         4      0   100%
	 ------------------------------------------------
	 TOTAL                          419    149    64%

…spot-support-oath

sherifnada

LGTM - is there a ticket to populate the streams with data?

airbyte-integrations/connectors/source-hubspot/source_hubspot/spec.json

airbyte-integrations/connectors/source-hubspot/source_hubspot/api.py

…spot-support-oath # Conflicts: # docs/integrations/sources/hubspot.md

htrueman · 2021-10-04T13:04:56Z

/test connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/1303677887
❌ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/1303677887
🐛 Binary file (standard input) matches

htrueman · 2021-10-04T13:58:48Z

/test connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/1303880674
✅ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/1303880674
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              200     94    53%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     24    41%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     111     11    90%
	 ------------------------------------------------------------------------
	 TOTAL                                                  856    416    51%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            83     59    29%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     19    42%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        660    393    40%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                         Stmts   Miss  Cover
	 ------------------------------------------------
	 main_dev.py                      6      6     0%
	 source_hubspot/__init__.py       2      0   100%
	 source_hubspot/api.py          364    131    64%
	 source_hubspot/client.py        37     12    68%
	 source_hubspot/errors.py         6      0   100%
	 source_hubspot/source.py         4      0   100%
	 ------------------------------------------------
	 TOTAL                          419    149    64%

sherifnada · 2021-10-16T00:47:06Z

@htrueman why didn't we publish the connector in this PR?

* Add required scopes to hubspot.md docs * Update spec with oauth creds. Update acceptance test config with oauth tests. Add oauth test creds configs. * Fix spec.js `oneOf` usage * Add CDK authenticators usage. Update source acceptance tests. Update spec.json. * Fix source unit tests * Update hubspot.md scopes docs * Add spec field titles. Update comments. * Update spec.json oauthFlowOutputParameters. * Update spec.json to pass SAT.

htrueman added 2 commits September 6, 2021 20:27

Add required scopes to hubspot.md docs

47dc57a

Update spec with oauth creds.

26598a1

Update acceptance test config with oauth tests. Add oauth test creds configs.

htrueman requested review from avida, Zirochkaa and yevhenii-ldv September 27, 2021 16:17

github-actions bot added the area/connectors Connector related issues label Sep 27, 2021

htrueman changed the title ~~Update spec with oauth creds.~~ 🎉 Source Hubspot: Support OAuth Sep 27, 2021

jrhizor temporarily deployed to more-secrets September 27, 2021 16:21 Inactive

htrueman mentioned this pull request Sep 27, 2021

Add required scopes to hubspot.md docs #5864

Closed

38 tasks

Merge remote-tracking branch 'origin/master' into htrueman/source-hub…

9370c7f

…spot-support-oath

Fix spec.js oneOf usage

1bb2d40

htrueman temporarily deployed to more-secrets September 27, 2021 18:04 Inactive

jrhizor temporarily deployed to more-secrets September 27, 2021 18:06 Inactive

avida suggested changes Sep 27, 2021

View reviewed changes

Zirochkaa suggested changes Sep 27, 2021

View reviewed changes

jrhizor temporarily deployed to more-secrets September 28, 2021 06:29 Inactive

Add CDK authenticators usage.

a0fe2ae

Update source acceptance tests. Update spec.json.

github-actions bot added the area/documentation Improvements or additions to documentation label Sep 28, 2021

htrueman temporarily deployed to more-secrets September 28, 2021 15:16 Inactive

jrhizor temporarily deployed to more-secrets September 28, 2021 15:16 Inactive

htrueman requested review from avida and Zirochkaa September 28, 2021 15:21

jrhizor temporarily deployed to more-secrets September 28, 2021 15:23 Inactive

Fix source unit tests

036e421

htrueman temporarily deployed to more-secrets September 28, 2021 15:43 Inactive

jrhizor temporarily deployed to more-secrets September 28, 2021 15:44 Inactive

avida approved these changes Sep 28, 2021

View reviewed changes

avida requested a review from sherifnada September 28, 2021 15:53

Zirochkaa approved these changes Sep 28, 2021

View reviewed changes

sherifnada reviewed Sep 29, 2021

View reviewed changes

airbyte-integrations/connectors/source-hubspot/source_hubspot/spec.json Outdated Show resolved Hide resolved

airbyte-integrations/connectors/source-hubspot/source_hubspot/api.py Outdated Show resolved Hide resolved

htrueman added 2 commits September 29, 2021 16:42

Merge remote-tracking branch 'origin/htrueman/source-hubspot-add-requ…

daf7419

…ired-scopes-doc' into htrueman/source-hubspot-support-oath

Update hubspot.md scopes docs

a84b3fe

htrueman temporarily deployed to more-secrets September 29, 2021 13:47 Inactive

htrueman mentioned this pull request Sep 29, 2021

Source Hubspot: migrate to CDK #5114

Closed

Add spec field titles.

02e01d7

Update comments.

htrueman temporarily deployed to more-secrets September 29, 2021 14:59 Inactive

Merge remote-tracking branch 'origin/master' into htrueman/source-hub…

93049e7

…spot-support-oath

jrhizor temporarily deployed to more-secrets September 29, 2021 15:37 Inactive

htrueman requested a review from sherifnada September 29, 2021 20:06

sherifnada approved these changes Oct 2, 2021

View reviewed changes

airbyte-integrations/connectors/source-hubspot/source_hubspot/spec.json Outdated Show resolved Hide resolved

airbyte-integrations/connectors/source-hubspot/source_hubspot/api.py Show resolved Hide resolved

Update spec.json oauthFlowOutputParameters.

8567580

htrueman temporarily deployed to more-secrets October 4, 2021 13:02 Inactive

Merge remote-tracking branch 'origin/master' into htrueman/source-hub…

610c3ec

…spot-support-oath # Conflicts: # docs/integrations/sources/hubspot.md

htrueman temporarily deployed to more-secrets October 4, 2021 13:06 Inactive

jrhizor temporarily deployed to more-secrets October 4, 2021 13:06 Inactive

htrueman mentioned this pull request Oct 4, 2021

🎉 Source Hubspot: Fill all implemented streams with data #6711

Closed

Update spec.json to pass SAT.

fc47028

htrueman temporarily deployed to more-secrets October 4, 2021 14:00 Inactive

jrhizor temporarily deployed to more-secrets October 4, 2021 14:01 Inactive

htrueman merged commit 351e998 into master Oct 4, 2021

htrueman deleted the htrueman/source-hubspot-support-oath branch October 4, 2021 14:11

jrhizor mentioned this pull request Oct 4, 2021

Bump Airbyte version from 0.30.9-alpha to 0.30.10-alpha #6741

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source Hubspot: Support OAuth #6465

🎉 Source Hubspot: Support OAuth #6465

htrueman commented Sep 27, 2021

CLAassistant commented Sep 27, 2021 •

edited

Loading

sherifnada Sep 28, 2021

htrueman Sep 29, 2021

htrueman Oct 4, 2021

htrueman commented Sep 29, 2021 •

edited by github-actions bot

Loading

sherifnada left a comment

htrueman commented Oct 4, 2021 •

edited by github-actions bot

Loading

htrueman commented Oct 4, 2021 •

edited by github-actions bot

Loading

sherifnada commented Oct 16, 2021

🎉 Source Hubspot: Support OAuth #6465

🎉 Source Hubspot: Support OAuth #6465

Conversation

htrueman commented Sep 27, 2021

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

CLAassistant commented Sep 27, 2021 • edited Loading

sherifnada Sep 28, 2021

Choose a reason for hiding this comment

htrueman Sep 29, 2021

Choose a reason for hiding this comment

htrueman Oct 4, 2021

Choose a reason for hiding this comment

htrueman commented Sep 29, 2021 • edited by github-actions bot Loading

sherifnada left a comment

Choose a reason for hiding this comment

htrueman commented Oct 4, 2021 • edited by github-actions bot Loading

htrueman commented Oct 4, 2021 • edited by github-actions bot Loading

sherifnada commented Oct 16, 2021

CLAassistant commented Sep 27, 2021 •

edited

Loading

htrueman commented Sep 29, 2021 •

edited by github-actions bot

Loading

htrueman commented Oct 4, 2021 •

edited by github-actions bot

Loading

htrueman commented Oct 4, 2021 •

edited by github-actions bot

Loading