Skip to content

update some connectors to use YAML specs - #12230

Merged
Pedro S. Lopez (pedroslopez) merged 14 commits into
masterfrom
pedroslopez/yaml-spec-connectors
Apr 21, 2022
Merged

Pedro S. Lopez (pedroslopez) merged 14 commits into
masterfrom
pedroslopez/yaml-spec-connectors

Conversation

@pedroslopez

@pedroslopez Pedro S. Lopez (pedroslopez) commented Apr 20, 2022 •

Copy link
Copy Markdown
Contributor

What

#11668 introduced yaml spec support, now let's actually use it!

How

Update 5 connectors to use a spec.yaml:

  • stripe
  • salesforce
  • hubspot
  • google sheets
  • exchange-rates

I basically ran the old spec.json files through https://codebeautify.org/json-to-yaml and made minor tweaks (mainly change single quotes to double quotes)

Also updated some references to the old files and bumped connector versions.
Doc updates coming soon w/ references to new spec.yaml files.

@pedroslopez Pedro S. Lopez (pedroslopez) linked an issue Apr 20, 2022 that may be closed by this pull request
@github-actions github-actions Bot added the area/connectors Connector related issues label Apr 20, 2022
@pedroslopez

Pedro S. Lopez (pedroslopez) commented Apr 20, 2022 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-stripe

🕑 connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/2198945566
✅ connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/2198945566
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  886    259    71%
Name                        Stmts   Miss  Cover
-----------------------------------------------
source_stripe/__init__.py       2      0   100%
source_stripe/streams.py      235     66    72%
source_stripe/source.py        22     11    50%
-----------------------------------------------
TOTAL                         259     77    70%

@pedroslopez

Pedro S. Lopez (pedroslopez) commented Apr 20, 2022 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2198946783
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2198946783
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  886    259    71%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/__init__.py            2      0   100%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/api.py               150     19    87%
source_salesforce/streams.py           282     60    79%
source_salesforce/rate_limiting.py      22      6    73%
source_salesforce/source.py             75     33    56%
source_salesforce/utils.py               8      7    12%
--------------------------------------------------------
TOTAL                                  547    126    77%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/utils.py               8      0   100%
source_salesforce/__init__.py            2      0   100%
source_salesforce/source.py             75      6    92%
source_salesforce/api.py               150     14    91%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/rate_limiting.py      22      3    86%
source_salesforce/streams.py           282     41    85%
--------------------------------------------------------
TOTAL                                  547     65    88%

@pedroslopez

Pedro S. Lopez (pedroslopez) commented Apr 21, 2022 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2199015565
❌ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2199015565
🐛 https://gradle.com/s/3qyv5nydwzd3g
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_oneof_usage[inputs0] - AssertionError: Th...
=================== 1 failed, 26 passed in 118.06s (0:01:58) ===================

@pedroslopez

Pedro S. Lopez (pedroslopez) commented Apr 21, 2022 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-google-sheets

🕑 connectors/source-google-sheets https://github.com/airbytehq/airbyte/actions/runs/2199043935
✅ connectors/source-google-sheets https://github.com/airbytehq/airbyte/actions/runs/2199043935
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  886    259    71%
Name                                                Stmts   Miss  Cover
-----------------------------------------------------------------------
google_sheets_source/models/spreadsheet_values.py      12      0   100%
google_sheets_source/models/spreadsheet.py             34      0   100%
google_sheets_source/models/__init__.py                 2      0   100%
google_sheets_source/__init__.py                        2      0   100%
google_sheets_source/helpers.py                       139     26    81%
google_sheets_source/client.py                         22      5    77%
google_sheets_source/google_sheets_source.py          103     84    18%
-----------------------------------------------------------------------
TOTAL                                                 314    115    63%

Python short test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
================== 22 passed, 1 skipped in 121.90s (0:02:01) ===================

@pedroslopez

Pedro S. Lopez (pedroslopez) commented Apr 21, 2022 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-exchange-rates

🕑 connectors/source-exchange-rates https://github.com/airbytehq/airbyte/actions/runs/2199103908
✅ connectors/source-exchange-rates https://github.com/airbytehq/airbyte/actions/runs/2199103908
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  886    259    71%

Python short test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
======================== 19 passed, 2 skipped in 16.33s ========================

@pedroslopez

Pedro S. Lopez (pedroslopez) commented Apr 21, 2022 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2199117927
✅ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2199117927
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  886    259    71%
Name                         Stmts   Miss  Cover
------------------------------------------------
source_hubspot/errors.py         6      0   100%
source_hubspot/__init__.py       2      0   100%
source_hubspot/streams.py      707     56    92%
source_hubspot/source.py        69     12    83%
------------------------------------------------
TOTAL                          784     68    91%

packages=find_packages(),
install_requires=MAIN_REQUIREMENTS,
package_data={"": ["*.json", "schemas/*.json", "schemas/shared/*.json"]},
package_data={"": ["*.json", "*.yaml", "schemas/*.json", "schemas/shared/*.json"]},

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: not adding *.yaml here weirdly didn't seem to cause any issues 🤔

@@ -0,0 +1,47 @@
documentationUrl: https://docs.airbyte.io/integrations/sources/stripe
connectionSpecification:
$schema: http://json-schema.org/draft-07/schema#

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feels a little odd to be doing this in a yaml file, but I think it's still right?

type: string
title: Credentials Title
description: Name of the credentials set
const: OAuth Credentials

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is slightly different than the spec.json counterpart: previously default and enum were defined, but it looks like that's not necessary anymore and was causing test failures.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup! Thanks for the change


MAIN_REQUIREMENTS = [
"airbyte-cdk~=0.1.49",
"airbyte-cdk~=0.1.55",

@pedroslopez Pedro S. Lopez (pedroslopez) Apr 21, 2022 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just pointing out that I noticed only hubspot has a patch version in the requirements. Others either don't have a version at all, or are set to ~=0.1

@pedroslopez

Pedro S. Lopez (pedroslopez) commented Apr 21, 2022 •

Copy link
Copy Markdown
Contributor Author

/publish connector=connectors/source-stripe

🕑 connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/2199283730
🚀 Successfully published connectors/source-stripe
❌ Couldn't auto-bump version for connectors/source-stripe

@pedroslopez

Pedro S. Lopez (pedroslopez) commented Apr 21, 2022 •

Copy link
Copy Markdown
Contributor Author

/publish connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2199285286
🚀 Successfully published connectors/source-salesforce
🚀 Auto-bumped version for connectors/source-salesforce
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2199285286

@octavia-squidington-iii
Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets April 21, 2022 02:06 Inactive
@octavia-squidington-iii
Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets April 21, 2022 02:06 Inactive
@codecov

codecov Bot commented Apr 21, 2022 •

Copy link
Copy Markdown

Codecov Report

❗ No coverage uploaded for pull request base (master@53799cb). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #12230   +/-   ##
=========================================
  Coverage          ?   82.93%           
=========================================
  Files             ?       21           
  Lines             ?     1904           
  Branches          ?        0           
=========================================
  Hits              ?     1579           
  Misses            ?      325           
  Partials          ?        0           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 53799cb...16bc799. Read the comment docs.

@pedroslopez

Pedro S. Lopez (pedroslopez) commented Apr 21, 2022 •

Copy link
Copy Markdown
Contributor Author

/publish connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2199353483
🚀 Successfully published connectors/source-hubspot
🚀 Auto-bumped version for connectors/source-hubspot
✅ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2199353483

@pedroslopez

Pedro S. Lopez (pedroslopez) commented Apr 21, 2022 •

Copy link
Copy Markdown
Contributor Author

/publish connector=connectors/source-google-sheets auto-bump-version=false

🕑 connectors/source-google-sheets https://github.com/airbytehq/airbyte/actions/runs/2199356387
🚀 Successfully published connectors/source-google-sheets
✅ connectors/source-google-sheets https://github.com/airbytehq/airbyte/actions/runs/2199356387

@pedroslopez

Pedro S. Lopez (pedroslopez) commented Apr 21, 2022 •

Copy link
Copy Markdown
Contributor Author

/publish connector=connectors/source-exchange-rates auto-bump-version=false

🕑 connectors/source-exchange-rates https://github.com/airbytehq/airbyte/actions/runs/2199357564
🚀 Successfully published connectors/source-exchange-rates
✅ connectors/source-exchange-rates https://github.com/airbytehq/airbyte/actions/runs/2199357564

@octavia-squidington-iii
Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets April 21, 2022 02:28 Inactive
@octavia-squidington-iii
Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets April 21, 2022 02:28 Inactive
@pedroslopez
Pedro S. Lopez (pedroslopez) temporarily deployed to more-secrets April 21, 2022 02:37 Inactive
@pedroslopez
Pedro S. Lopez (pedroslopez) temporarily deployed to more-secrets April 21, 2022 02:37 Inactive
@pedroslopez
Pedro S. Lopez (pedroslopez) deleted the pedroslopez/yaml-spec-connectors branch April 21, 2022 02:59
Yevhen Sukhomud (suhomud) pushed a commit that referenced this pull request May 23, 2022
* salesforce uses spec.yaml

* stripe uses spec.yaml

* revert unintended changes

* bump stripe version

* add yaml files to package_data

* hubspot uses a spec.yaml

* use double quotes in yaml files

* google sheets uses spec.yaml

* exhcange-rates uses spec.yaml

* remove usage of default and enum in spec

* bump hubspot cdk requirement

* auto-bump connector version

* auto-bump connector version

* manually bump versions

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

This branch was previously deployed

1 inactive deployment
more-secrets — 16bc7991 Deployed Apr 21, 2022 by pedroslopez
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connectors Connector related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update 5 connectors to use YAML spec

3 participants