🎉 New Source: Hubplanner #12145

RickRen7575 · 2022-04-19T17:09:07Z

What

I added a source connector for the Hubplanner API. Here is the information on the Hubplanner API: https://github.com/hubplanner/API

How

I utilized the generate.sh script to provide a scaffold and then followed the YouTube video here: https://www.youtube.com/watch?v=kJ3hLoNfz_E&list=PLgyvStszwUHjb-t_wfGizz1VZgwX7Td5b&index=2

I also took inspiration from the Singer tap for Hubplanner: https://github.com/rangle/tap-hubplanner

🚨 User Impact 🚨

No breaking changes.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Community member

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

I tested running check, discover, and read commands with the configured_catalog.json file provided, and it worked. I also tested creating the source in the UI and creating a connection that loads Hubplanner data to BigQuery and I also tested loading to a local JSON file. Both worked!

CLAassistant · 2022-04-19T17:09:12Z

All committers have signed the CLA.

RickRen7575 · 2022-04-19T17:15:52Z

I still need to add integration and unit tests, but I just wanted to open the PR first

RickRen7575 · 2022-04-19T19:56:38Z

@marcosmarxm what is a good example of unit/integration test suite comparable to Hubplanner? Looks like some connectors implement tests and others don't, so just want to know what is best to compare against and structure mine

RickRen7575 · 2022-04-21T16:51:00Z

airbyte-integrations/connectors/source-hubplanner/unit_tests/test_source.py

+def test_check_connection(mocker):
+    source = SourceHubplanner()
+    logger_mock, config_mock = MagicMock(), MagicMock()
+    assert source.check_connection(logger_mock, config_mock) == (True, None)


@marcosmarxm for example, I am a little confused how I am supposed to set this up to pass. All the other connectors look like they are setting up fixtures that just have like "api_key": "test_api_key" as their config. I don't understand why something like that would pass without a valid config.

The goal of this test is to assert the behavior of check_connection on successful or failed connection response by the API. You should use request_mock to forge mock success and failure request and assert on the behavior of check_connection.

@alafanechere so I would copy basically an example of a "failed response" from the hubplanner API, set up a test to mock that as the response, and make sure that check_connection fails in that instance?

alafanechere · 2022-04-22T18:01:40Z

Hi @RickRen7575 thank you for this contribution. We need to create a sandbox Hubplanner account to run tests on your connectors. I'll go for a review once we have this account.

alafanechere

Could you please:

Format your code with ./gradlew format
Run the acceptance tests and share their output:
./gradlew :airbyte-integrations:connectors:source-hubplanner:integrationTest (this is a built-in test suite that is configured in acceptance-test-config.yml)

RickRen7575 · 2022-04-22T19:29:20Z

That would be awesome!!

alafanechere · 2022-04-25T13:27:22Z

Format your code with ./gradlew format

Run the acceptance tests and share their output:
./gradlew :airbyte-integrations:connectors:source-hubplanner:integrationTest (this is a built-in test suite that is configured in acceptance-test-config.yml)

@RickRen7575 are you ready to perform these tasks? I won't be able to do this myself as I've no maintainer permission on your organization's fork.

alafanechere · 2022-05-09T08:37:22Z

Hey @RickRen7575,
I'm closing this PR because I did not receive updates from your side and I don't have permission to push the changes I requested. Feel free to re-open whenever you like, we now have a sandbox account for Hubplanner to test your connector and make this PR move forward.

RickRen7575 · 2022-05-16T12:53:33Z

@alafanechere sorry for the delay - I've been out for the last 3 weeks. I'll run these things and then push updates!

RickRen7575 · 2022-05-16T13:21:50Z

@alafanechere I don't see the ability to reopen this PR. Do you mind doing that on your end?

alafanechere · 2022-05-16T14:16:25Z

@RickRen7575 I reopened it!

RickRen7575 · 2022-05-16T15:23:55Z

@alafanechere I just tried running the integration tests and got this output:
`

Task :airbyte-integrations:bases:source-acceptance-test:installLocalReqs
[python] cmd /c .venv\Scripts\python.exe -m pip install .[dev,tests]
Processing c:\users\rickyrenner\documents\repos\airbyte\airbyte\airbyte-integrations\bases\source-acceptance-test
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
WARNING: source-acceptance-test 0.0.0 does not provide the extra 'dev'
WARNING: source-acceptance-test 0.0.0 does not provide the extra 'tests'
Collecting airbyte-cdk~=0.1.56
Using cached airbyte_cdk-0.1.56-py3-none-any.whl (83 kB)
Collecting docker~=5.0.3
Using cached docker-5.0.3-py2.py3-none-any.whl (146 kB)
Collecting PyYAML~=5.4
Using cached PyYAML-5.4.1-cp310-cp310-win_amd64.whl
Collecting icdiff~=1.9
Using cached icdiff-1.9.1.tar.gz (9.1 kB)
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Collecting inflection~=0.5
Using cached inflection-0.5.1-py2.py3-none-any.whl (9.5 kB)
Collecting pdbpp~=0.10
Using cached pdbpp-0.10.3-py2.py3-none-any.whl (23 kB)
Collecting pydantic~=1.6
Using cached pydantic-1.9.0-cp310-cp310-win_amd64.whl (2.1 MB)
Requirement already satisfied: pytest~=6.1 in c:\users\rickyrenner\documents\repos\airbyte\airbyte\airbyte-integrations\bases\source-acceptance-test.venv\lib\site-packages (from source-acceptance-test==0.0.0) (6.1.2)
Collecting pytest-sugar~=0.9
Using cached pytest-sugar-0.9.4.tar.gz (12 kB)
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Collecting pytest-timeout~=1.4
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pprintpp~=0.4
Using cached pprintpp-0.4.0-py2.py3-none-any.whl (16 kB)
Collecting dpath~=2.0.1
Using cached dpath-2.0.6-py3-none-any.whl (15 kB)
Collecting jsonschema~=3.2.0
Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Collecting jsonref==0.2
Using cached jsonref-0.2-py3-none-any.whl (9.3 kB)
Collecting pendulum
Using cached pendulum-2.1.2-cp310-cp310-win_amd64.whl
Collecting requests
Using cached requests-2.27.1-py2.py3-none-any.whl (63 kB)
Collecting vcrpy
Using cached vcrpy-4.1.1-py2.py3-none-any.whl (40 kB)
Collecting sentry-sdk~=1.5.1
Using cached sentry_sdk-1.5.12-py2.py3-none-any.whl (145 kB)
Collecting backoff
Using cached backoff-2.0.1-py3-none-any.whl (14 kB)
Collecting Deprecated~=1.2
Using cached Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.3.2-py3-none-any.whl (54 kB)
INFO: pip is looking at multiple versions of to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of airbyte-cdk to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of jsonref to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of source-acceptance-test[dev,tests] to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement pywin32==227; sys_platform == "win32" (from docker) (from versions: 302, 303, 304)
ERROR: No matching distribution found for pywin32==227; sys_platform == "win32"
WARNING: You are using pip version 21.3.1; however, version 22.1 is available.
You should consider upgrading via the 'C:\Users\RickyRenner\Documents\Repos\airbyte\airbyte\airbyte-integrations\bases\source-acceptance-test.venv\Scripts\python.exe -m pip install --upgrade pip' command.

Task :airbyte-integrations:bases:source-acceptance-test:installLocalReqs FAILED`

alafanechere · 2022-05-16T20:31:30Z

Are you using WSL to run this? Could you check the alternative pytest commands in the generated README file to run the acceptance tests?

RickRen7575 · 2022-05-17T12:26:09Z

I was trying to use just straight up Windows, but I can try WSL instead!

alafanechere · 2022-06-07T17:30:26Z

@RickRen7575 do you manage to make the test pass with WSL?

RickRen7575 · 2022-06-07T17:32:20Z

Hi @alafanechere I'm sorry but I've just been swamped recently. I hope I can get back on this soon. I tried running the tests and ran into an issue and then got sidetracked.

marcosmarxm · 2022-08-10T20:15:20Z

Thanks for the contribution @RickRen7575 I merged the code in #15521

RickRen7575 · 2022-08-11T12:09:34Z

@marcosmarxm oh great!

RickRen7575 added 5 commits April 15, 2022 13:24

added hubplanner source connector

a9ea5e3

Merge branch 'airbytehq:master' into master

bd985c5

feat: added more streams to read from

82f9d5d

Merge branch 'airbytehq:master' into master

a74f450

cleaned up some unneeded integration tests

62f84a6

github-actions bot added the area/connectors Connector related issues label Apr 19, 2022

octavia-squidington-iii added the community label Apr 19, 2022

RickRen7575 changed the title ~~Hubplanner Source Connector~~ New Source Connector: Hubplanner using CDK Apr 19, 2022

Merge branch 'airbytehq:master' into master

1adcaa2

RickRen7575 commented Apr 21, 2022

View reviewed changes

alafanechere self-assigned this Apr 22, 2022

alafanechere requested changes Apr 22, 2022

View reviewed changes

marcosmarxm added the blocked label May 5, 2022

alafanechere closed this May 9, 2022

Merge branch 'airbytehq:master' into master

fad7296

alafanechere reopened this May 16, 2022

bleonard added the bounty label Jun 2, 2022

bleonard added the connectors/source/hubplanner label Jun 2, 2022

alafanechere added internal and removed blocked bounty labels Jun 7, 2022

alafanechere assigned natalyjazzviolin and unassigned natalyjazzviolin and alafanechere Jul 11, 2022

marcosmarxm changed the title ~~New Source Connector: Hubplanner using CDK~~ 🎉 New Source: Hubplanner Aug 4, 2022

marcosmarxm mentioned this pull request Aug 10, 2022

🎉 New Source: Hubplanner #15521

Merged

37 tasks

marcosmarxm closed this Aug 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: Hubplanner #12145

🎉 New Source: Hubplanner #12145

RickRen7575 commented Apr 19, 2022 •

edited

Loading

CLAassistant commented Apr 19, 2022 •

edited

Loading

RickRen7575 commented Apr 19, 2022

RickRen7575 commented Apr 19, 2022

RickRen7575 Apr 21, 2022

alafanechere Apr 22, 2022

RickRen7575 Apr 22, 2022

alafanechere commented Apr 22, 2022

alafanechere left a comment •

edited

Loading

RickRen7575 commented Apr 22, 2022

alafanechere commented Apr 25, 2022

alafanechere commented May 9, 2022

RickRen7575 commented May 16, 2022

RickRen7575 commented May 16, 2022

alafanechere commented May 16, 2022

RickRen7575 commented May 16, 2022

alafanechere commented May 16, 2022

RickRen7575 commented May 17, 2022

alafanechere commented Jun 7, 2022

RickRen7575 commented Jun 7, 2022

marcosmarxm commented Aug 10, 2022

RickRen7575 commented Aug 11, 2022

🎉 New Source: Hubplanner #12145

🎉 New Source: Hubplanner #12145

Conversation

RickRen7575 commented Apr 19, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

CLAassistant commented Apr 19, 2022 • edited Loading

RickRen7575 commented Apr 19, 2022

RickRen7575 commented Apr 19, 2022

RickRen7575 Apr 21, 2022

Choose a reason for hiding this comment

alafanechere Apr 22, 2022

Choose a reason for hiding this comment

RickRen7575 Apr 22, 2022

Choose a reason for hiding this comment

alafanechere commented Apr 22, 2022

alafanechere left a comment • edited Loading

Choose a reason for hiding this comment

RickRen7575 commented Apr 22, 2022

alafanechere commented Apr 25, 2022

alafanechere commented May 9, 2022

RickRen7575 commented May 16, 2022

RickRen7575 commented May 16, 2022

alafanechere commented May 16, 2022

RickRen7575 commented May 16, 2022

alafanechere commented May 16, 2022

RickRen7575 commented May 17, 2022

alafanechere commented Jun 7, 2022

RickRen7575 commented Jun 7, 2022

marcosmarxm commented Aug 10, 2022

RickRen7575 commented Aug 11, 2022

RickRen7575 commented Apr 19, 2022 •

edited

Loading

CLAassistant commented Apr 19, 2022 •

edited

Loading

alafanechere left a comment •

edited

Loading