🎉 Source Zuora: Add `OAuth` support for Airbyte Cloud, Migrated to new CI Sandbox #6575

bazarnov · 2021-09-30T14:52:58Z

What

#6298 - Source Zuora: support oauth

How

edited spec.json
added more endpoints to connect to (as options in UI)
migrated to new CI Sandbox (airbyte-integration)
added support of streams without any cursor (updatedDate, createdDate) - for such cases, we fetch the whole object using Full-Refresh mode.
edited integration-test.py

Pre-merge Checklist

Updating a connector

Community member or Airbyter

Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions
Connector version bumped like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

antixar

are you sure that you added oauth properties to the spec file? I see changes for sandbox options only.

airbyte-integrations/connectors/source-zuora/source_zuora/spec.json

airbyte-integrations/connectors/source-zuora/source_zuora/zuora_endpoint.py

bazarnov · 2021-10-04T09:46:11Z

are you sure that you added oauth properties to the spec file? I see changes for sandbox options only.

Haven't manage to submit the latest changes yet) The tests on new CI Sandbox require to update some parts of the connector.

bazarnov · 2021-10-05T11:49:13Z

/test connector=connectors/source-zuora

🕑 connectors/source-zuora https://github.com/airbytehq/airbyte/actions/runs/1307677875
❌ connectors/source-zuora https://github.com/airbytehq/airbyte/actions/runs/1307677875
🐛 https://gradle.com/s/srrubn5ho7xlk
🕑 connectors/source-zuora https://github.com/airbytehq/airbyte/actions/runs/1307677875
✅ connectors/source-zuora https://github.com/airbytehq/airbyte/actions/runs/1307677875
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              197     93    53%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     24    41%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     111     11    90%
	 ------------------------------------------------------------------------
	 TOTAL                                                  853    415    51%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                     Stmts   Miss  Cover
	 ------------------------------------------------------------
	 source_zuora/__init__.py                     2      0   100%
	 source_zuora/source.py                     207     47    77%
	 source_zuora/zuora_auth.py                  10      0   100%
	 source_zuora/zuora_endpoint.py               4      0   100%
	 source_zuora/zuora_errors.py                24     10    58%
	 source_zuora/zuora_excluded_streams.py       2      0   100%
	 ------------------------------------------------------------
	 TOTAL                                      249     57    77%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                     Stmts   Miss  Cover
	 ------------------------------------------------------------
	 source_zuora/__init__.py                     2      2     0%
	 source_zuora/source.py                     207    207     0%
	 source_zuora/zuora_auth.py                  10     10     0%
	 source_zuora/zuora_endpoint.py               4      4     0%
	 unit_tests/unit_test.py::test_example �[32mPASSED�[0m/actions-runner/_work/airbyte/airbyte/airbyte-integrations/connectors/source-zuora/.venv/lib/python3.8/site-packages/coverage/control.py:761: CoverageWarning: No data was collected. (no-data-collected)
	   self._warn("No data was collected.", slug="no-data-collected")
	 source_zuora/zuora_errors.py                24     24     0%
	 source_zuora/zuora_excluded_streams.py       2      2     0%
	 ------------------------------------------------------------
	 TOTAL                                      249    249     0%

antixar

looks good!

airbyte-integrations/connectors/source-zuora/source_zuora/spec.json

…full_refresh streams

bazarnov · 2021-10-06T08:11:59Z

/test connector=connectors/source-zuora

🕑 connectors/source-zuora https://github.com/airbytehq/airbyte/actions/runs/1311005112
✅ connectors/source-zuora https://github.com/airbytehq/airbyte/actions/runs/1311005112
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              197     93    53%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     24    41%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     111     11    90%
	 ------------------------------------------------------------------------
	 TOTAL                                                  853    415    51%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                     Stmts   Miss  Cover
	 ------------------------------------------------------------
	 source_zuora/__init__.py                     2      0   100%
	 source_zuora/source.py                     208     48    77%
	 source_zuora/zuora_auth.py                  10      0   100%
	 source_zuora/zuora_endpoint.py               4      0   100%
	 source_zuora/zuora_errors.py                24     10    58%
	 source_zuora/zuora_excluded_streams.py       2      0   100%
	 ------------------------------------------------------------
	 TOTAL                                      250     58    77%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                     Stmts   Miss  Cover
	 ------------------------------------------------------------
	 source_zuora/__init__.py                     2      2     0%
	 source_zuora/source.py                     208    208     0%
	 source_zuora/zuora_auth.py                  10     10     0%
	 source_zuora/zuora_endpoint.py               4      4     0%
	 source_zuora/zuora_errors.py                24     24     0%
	 source_zuora/zuora_excluded_streams.py       2      2     0%
	 ------------------------------------------------------------
	 TOTAL                                      250    250     0%

bazarnov · 2021-10-06T12:00:41Z

/publish connector=connectors/source-zuora

🕑 connectors/source-zuora https://github.com/airbytehq/airbyte/actions/runs/1311763819
✅ connectors/source-zuora https://github.com/airbytehq/airbyte/actions/runs/1311763819

…w CI Sandbox (airbytehq#6575) * edited spec.json * added more endpoints to connect to (as options in UI) * migrated to new CI Sandbox (airbyte-integration) * added support of streams without any cursor (updatedDate, createdDate) - for such cases, we fetch the whole object * * using Full-Refresh mode. * edited integration-test.py

added more endpoints, corrected integration_test.py, bumped vesion

d04655f

bazarnov self-assigned this Sep 30, 2021

github-actions bot added the area/connectors Connector related issues label Sep 30, 2021

added changelog

98bf5d1

bazarnov linked an issue Sep 30, 2021 that may be closed by this pull request

Source Zuora: support oauth #6298

Closed

github-actions bot added the area/documentation Improvements or additions to documentation label Sep 30, 2021

bazarnov temporarily deployed to more-secrets September 30, 2021 14:54 Inactive

bazarnov requested review from antixar and lazebnyi October 1, 2021 11:20

jrhizor temporarily deployed to more-secrets October 1, 2021 11:21 Inactive

bazarnov linked an issue Oct 2, 2021 that may be closed by this pull request

Use new Zuora Sandbox in CI #6035

Closed

antixar suggested changes Oct 4, 2021

View reviewed changes

updated connector and tests for new CI Sandbox + Oauth Flow is added

b6679c2

bazarnov requested a review from antixar October 4, 2021 13:02

bazarnov temporarily deployed to more-secrets October 4, 2021 13:03 Inactive

added miliseconds to parsed datetime string

76a6faa

bazarnov temporarily deployed to more-secrets October 4, 2021 14:06 Inactive

jrhizor temporarily deployed to more-secrets October 4, 2021 14:07 Inactive

removed expected_records test

39d9c96

bazarnov temporarily deployed to more-secrets October 4, 2021 14:35 Inactive

jrhizor temporarily deployed to more-secrets October 4, 2021 14:36 Inactive

jrhizor temporarily deployed to more-secrets October 5, 2021 08:26 Inactive

added excluded streams + integration test for this

7a99d52

bazarnov temporarily deployed to more-secrets October 5, 2021 11:50 Inactive

jrhizor temporarily deployed to more-secrets October 5, 2021 11:51 Inactive

jrhizor temporarily deployed to more-secrets October 5, 2021 12:20 Inactive

antixar approved these changes Oct 5, 2021

View reviewed changes

airbyte-integrations/connectors/source-zuora/source_zuora/spec.json Outdated Show resolved Hide resolved

fixed full_refresh stream edge cases, removed updating of state from …

0c3561c

…full_refresh streams

bazarnov temporarily deployed to more-secrets October 5, 2021 18:06 Inactive

updated after review

58b3744

bazarnov temporarily deployed to more-secrets October 5, 2021 18:10 Inactive

jrhizor temporarily deployed to more-secrets October 5, 2021 18:12 Inactive

fixed invalid config validation

f175372

bazarnov temporarily deployed to more-secrets October 6, 2021 08:13 Inactive

jrhizor temporarily deployed to more-secrets October 6, 2021 08:14 Inactive

formated, added environements to documentation .md

f88df8c

bazarnov temporarily deployed to more-secrets October 6, 2021 09:17 Inactive

bazarnov temporarily deployed to more-secrets October 6, 2021 11:11 Inactive

jrhizor temporarily deployed to more-secrets October 6, 2021 12:03 Inactive

bazarnov temporarily deployed to more-secrets October 6, 2021 13:40 Inactive

bazarnov merged commit 55dae7f into master Oct 6, 2021

bazarnov deleted the bazarnov/6298-zuora-oauth branch October 6, 2021 14:10

jrhizor mentioned this pull request Oct 7, 2021

Bump Airbyte version from 0.30.15-alpha to 0.30.16-alpha #6880

Merged

lazebnyi removed their request for review May 30, 2022 10:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source Zuora: Add `OAuth` support for Airbyte Cloud, Migrated to new CI Sandbox #6575

🎉 Source Zuora: Add `OAuth` support for Airbyte Cloud, Migrated to new CI Sandbox #6575

bazarnov commented Sep 30, 2021 •

edited

Loading

antixar left a comment

bazarnov commented Oct 4, 2021

bazarnov commented Oct 5, 2021 •

edited by github-actions bot

Loading

antixar left a comment

bazarnov commented Oct 6, 2021 •

edited by github-actions bot

Loading

bazarnov commented Oct 6, 2021 •

edited by github-actions bot

Loading

🎉 Source Zuora: Add OAuth support for Airbyte Cloud, Migrated to new CI Sandbox #6575

🎉 Source Zuora: Add OAuth support for Airbyte Cloud, Migrated to new CI Sandbox #6575

Conversation

bazarnov commented Sep 30, 2021 • edited Loading

What

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

antixar left a comment

Choose a reason for hiding this comment

bazarnov commented Oct 4, 2021

bazarnov commented Oct 5, 2021 • edited by github-actions bot Loading

antixar left a comment

Choose a reason for hiding this comment

bazarnov commented Oct 6, 2021 • edited by github-actions bot Loading

bazarnov commented Oct 6, 2021 • edited by github-actions bot Loading

🎉 Source Zuora: Add `OAuth` support for Airbyte Cloud, Migrated to new CI Sandbox #6575

🎉 Source Zuora: Add `OAuth` support for Airbyte Cloud, Migrated to new CI Sandbox #6575

bazarnov commented Sep 30, 2021 •

edited

Loading

bazarnov commented Oct 5, 2021 •

edited by github-actions bot

Loading

bazarnov commented Oct 6, 2021 •

edited by github-actions bot

Loading

bazarnov commented Oct 6, 2021 •

edited by github-actions bot

Loading