🎉 New Source: Insightly [python cdk] #18164

henriblancke · 2022-10-19T15:17:41Z

What

New Source: Insightly

Notes

I have a sandbox account for integration testing, let me know how to pass those credentials along.

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Integration

Put your integration tests output here.

Acceptance

marcosmarxm

a few comments, but looks great! can you share the credentials with me (send DM in Slack if possible)

airbyte-integrations/connectors/source-insightly/integration_tests/catalog.json

airbyte-integrations/connectors/source-insightly/integration_tests/configured_catalog.json

airbyte-integrations/connectors/source-insightly/logs.log

marcosmarxm · 2022-10-19T17:21:25Z

airbyte-integrations/connectors/source-insightly/source_insightly/source.py

+    def check_connection(self, logger, config) -> Tuple[bool, any]:
+        try:
+            token = config.get("token")
+            response = requests.get(f"{BASE_URL}Instance", auth=(token,''))
+            response.raise_for_status()
+
+            result = response.json()
+            logger.info(result)
+
+            return True, None
+        except Exception as e:
+            return False, e


Can you check how other sources do the check_connection it works how you're doing but there are cleaner ways.

@marcosmarxm any specific connectors you want me to look at? This mainly aims to check if you have valid credentials. The /Instance endpoint returns information about your insightly instance. It's more generic than checking a single stream?

airbyte-integrations/connectors/source-insightly/unit_tests/test_incremental_streams.py

airbyte-integrations/connectors/source-insightly/unit_tests/test_source.py

marcosmarxm · 2022-10-20T17:38:22Z

@sajarin I'm reviewing this.

Signed-off-by: Henri Blancke <blanckehenri@gmail.com>

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

Signed-off-by: Henri Blancke <blanckehenri@gmail.com>

marcosmarxm · 2022-10-20T18:24:06Z

/test connector=connectors/source-insightly

🕑 connectors/source-insightly https://github.com/airbytehq/airbyte/actions/runs/3291987974
✅ connectors/source-insightly https://github.com/airbytehq/airbyte/actions/runs/3291987974
Python tests coverage:

Name                           Stmts   Miss  Cover
--------------------------------------------------
source_insightly/__init__.py       2      0   100%
source_insightly/source.py       211     43    80%
--------------------------------------------------
TOTAL                            213     43    80%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_incremental.py       145     20    86%   21-23, 29-31, 36-43, 48-61, 224
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     10    87%   15-16, 24-30, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1351    451    67%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
======================== 27 passed, 2 skipped in 24.88s ========================

marcosmarxm · 2022-10-20T18:47:24Z

/publish connector=connectors/source-insightly

🕑 Publishing the following connectors:
connectors/source-insightly
https://github.com/airbytehq/airbyte/actions/runs/3292130494

Connector	Did it publish?	Were definitions generated?
connectors/source-insightly	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

marcosmarxm · 2022-10-20T19:17:23Z

@RealChrisSean another hacktober contribution :)

RealChrisSean · 2022-10-21T23:14:24Z

Congrats @henriblancke!!!! :)

* [ADD] logic for source insightly Signed-off-by: Henri Blancke <blanckehenri@gmail.com> * [UPD] schema cleanup Signed-off-by: Henri Blancke <blanckehenri@gmail.com> * [ADD] documentation Signed-off-by: Henri Blancke <blanckehenri@gmail.com> * [UPD] configured catalog Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> * [RMV] catalog and logs Signed-off-by: Henri Blancke <blanckehenri@gmail.com> * [UPD] source tests Signed-off-by: Henri Blancke <blanckehenri@gmail.com> * format files * fix pk * add seed config * auto-bump connector version Signed-off-by: Henri Blancke <blanckehenri@gmail.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Oct 19, 2022

octavia-squidington-iv added bounty community connectors/source/insightly labels Oct 19, 2022

marcosmarxm requested changes Oct 19, 2022

View reviewed changes

henriblancke requested a review from marcosmarxm October 19, 2022 19:22

marcosmarxm added internal hacktober and removed bounty labels Oct 19, 2022

marcosmarxm changed the title ~~🎉 New Source: Insightly~~ 🎉 New Source: Insightly [python cdk[ Oct 19, 2022

marcosmarxm changed the title ~~🎉 New Source: Insightly [python cdk[~~ 🎉 New Source: Insightly [python cdk] Oct 19, 2022

henriblancke and others added 8 commits October 20, 2022 14:45

[ADD] logic for source insightly

25aa612

Signed-off-by: Henri Blancke <blanckehenri@gmail.com>

[UPD] schema cleanup

f4d4b53

Signed-off-by: Henri Blancke <blanckehenri@gmail.com>

[ADD] documentation

844ad5c

Signed-off-by: Henri Blancke <blanckehenri@gmail.com>

[UPD] configured catalog

a18f5e7

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

[RMV] catalog and logs

2403051

Signed-off-by: Henri Blancke <blanckehenri@gmail.com>

[UPD] source tests

4100837

Signed-off-by: Henri Blancke <blanckehenri@gmail.com>

format files

047b403

fix pk

2980c6e

marcosmarxm force-pushed the source-insightly branch from 696dfa0 to 2980c6e Compare October 20, 2022 18:23

add seed config

4d9909d

marcosmarxm approved these changes Oct 20, 2022

View reviewed changes

auto-bump connector version

f52b202

marcosmarxm merged commit f861a75 into airbytehq:master Oct 20, 2022

RealChrisSean added the python Pull requests that update Python code label Nov 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: Insightly [python cdk] #18164

🎉 New Source: Insightly [python cdk] #18164

henriblancke commented Oct 19, 2022 •

edited by RealChrisSean

marcosmarxm left a comment

marcosmarxm Oct 19, 2022

henriblancke Oct 19, 2022 •

edited

marcosmarxm commented Oct 20, 2022

marcosmarxm commented Oct 20, 2022 •

edited by github-actions bot

marcosmarxm commented Oct 20, 2022 •

edited by github-actions bot

marcosmarxm commented Oct 20, 2022

RealChrisSean commented Oct 21, 2022

🎉 New Source: Insightly [python cdk] #18164

🎉 New Source: Insightly [python cdk] #18164

Conversation

henriblancke commented Oct 19, 2022 • edited by RealChrisSean

What

Notes

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

marcosmarxm left a comment

Choose a reason for hiding this comment

marcosmarxm Oct 19, 2022

Choose a reason for hiding this comment

henriblancke Oct 19, 2022 • edited

Choose a reason for hiding this comment

marcosmarxm commented Oct 20, 2022

marcosmarxm commented Oct 20, 2022 • edited by github-actions bot

Build Passed

marcosmarxm commented Oct 20, 2022 • edited by github-actions bot

marcosmarxm commented Oct 20, 2022

RealChrisSean commented Oct 21, 2022

henriblancke commented Oct 19, 2022 •

edited by RealChrisSean

henriblancke Oct 19, 2022 •

edited

marcosmarxm commented Oct 20, 2022 •

edited by github-actions bot

marcosmarxm commented Oct 20, 2022 •

edited by github-actions bot