-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🎉 New source: Native Hubspot connector #2215
Merged
Merged
Changes from 15 commits
Commits
Show all changes
44 commits
Select commit
Hold shift + click to select a range
49cb4cb
#2150 Issue: created native connector with schema folder populated
5423d13
#2150 Issue: make format code
10a106b
first version
eugene-kulak e667fd0
fix few issues
eugene-kulak 10af6c9
fix issues
eugene-kulak 2e47519
fix read issue
eugene-kulak a180e71
format
eugene-kulak 21152c6
docs
eugene-kulak 3dc6556
docker tags
eugene-kulak 66b03d0
extend configured catalog for testing
eugene-kulak 2bc9f01
fix source definitions
eugene-kulak 0611ac7
format
eugene-kulak 42f24bb
fix call rate issue, add backoff for retry after
eugene-kulak 4479e02
add general backoff
eugene-kulak dd4e982
write secrets for new connector
eugene-kulak fc57695
drop singer connector registration
eugene-kulak a2ee039
refactor streams, resolve properties in schemas at runtime
eugene-kulak aa766eb
replace deprecated endpoint for company contacts
eugene-kulak 759043e
replace deprecated pipeline endpoint
eugene-kulak 4d6c851
update comments
eugene-kulak bca6024
update docs
eugene-kulak 3b65548
fix typo
eugene-kulak 39f0cf7
fix stream contact lists
eugene-kulak 136190a
fix pagination and forms result fetching
eugene-kulak f861888
fix health_check
eugene-kulak caf1c26
format and update catalog
eugene-kulak 6788be2
revert changes
eugene-kulak e084f46
drop singer based hubspot
eugene-kulak 8b52bc3
fix company contacts substream
eugene-kulak 0e52c9a
move deals to separate test
eugene-kulak 308cbec
fix deals tests
eugene-kulak a1f2b69
remove dynamic fields from records
eugene-kulak a6830f8
move deals to catalog again
eugene-kulak 94c6b75
extend CRMObjectStream with associations
eugene-kulak 36f4124
format
eugene-kulak 3a664eb
update schemas with updated field, change engagement layout
eugene-kulak 37f3acb
fix Campaign stream
eugene-kulak ad6988f
remove custom tests
eugene-kulak e00c32a
remove dependency
eugene-kulak 1312d21
remove oauth
eugene-kulak 0bfabee
Source Hubspot: add incremental streams (#2425)
keu 61c8805
Source Hubspot: best practices (#2537)
keu 2764dd3
Merge remote-tracking branch 'origin/master' into keu/source-hubspot-…
eugene-kulak a35389b
restored configured_catalog.json
eugene-kulak File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
7 changes: 7 additions & 0 deletions
7
...ain/resources/config/STANDARD_SOURCE_DEFINITION/36c891d9-4bd9-43ac-bad2-10e12756272c.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
{ | ||
"sourceDefinitionId": "36c891d9-4bd9-43ac-bad2-10e12756272c", | ||
"name": "Hubspot", | ||
"dockerRepository": "airbyte/source-hubspot", | ||
"dockerImageTag": "0.1.0", | ||
"documentationUrl": "https://https://docs.airbyte.io/integrations/sources/hubspot" | ||
} |
2 changes: 1 addition & 1 deletion
2
...ain/resources/config/STANDARD_SOURCE_DEFINITION/57eb1576-8f52-463d-beb6-2e107cdf571d.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
* | ||
!Dockerfile | ||
!Dockerfile.test | ||
!source_hubspot | ||
!setup.py | ||
!secrets |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
NEW_SOURCE_CHECKLIST.md |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
FROM airbyte/integration-base-python:dev | ||
|
||
# Bash is installed for more convenient debugging. | ||
RUN apt-get update && apt-get install -y bash && rm -rf /var/lib/apt/lists/* | ||
|
||
ENV CODE_PATH="source_hubspot" | ||
ENV AIRBYTE_IMPL_MODULE="source_hubspot" | ||
ENV AIRBYTE_IMPL_PATH="SourceHubspot" | ||
|
||
WORKDIR /airbyte/integration_code | ||
COPY $CODE_PATH ./$CODE_PATH | ||
COPY setup.py ./ | ||
RUN pip install ".[main]" | ||
|
||
LABEL io.airbyte.version=0.1.0 | ||
LABEL io.airbyte.name=airbyte/source-hubspot |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,64 @@ | ||
# Hubspot Source | ||
|
||
This is the repository for the Hubspot source connector, written in Python. | ||
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/hubspot). | ||
|
||
## Local development | ||
|
||
### Prerequisites | ||
**To iterate on this connector, make sure to complete this prerequisites section.** | ||
|
||
#### Build & Activate Virtual Environment | ||
First, build the module by running the following from the `airbyte` project root directory: | ||
``` | ||
./gradlew :airbyte-integrations:connectors:source-hubspot:build | ||
``` | ||
|
||
This will generate a virtualenv for this module in `source-hubspot/.venv`. Make sure this venv is active in your | ||
development environment of choice. To activate the venv from the terminal, run: | ||
``` | ||
cd airbyte-integrations/connectors/source-hubspot # cd into the connector directory | ||
source .venv/bin/activate | ||
``` | ||
If you are in an IDE, follow your IDE's instructions to activate the virtualenv. | ||
|
||
#### Create credentials | ||
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/hubspot) | ||
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_hubspot/spec.json` file. | ||
See `sample_files/sample_config.json` for a sample config file. | ||
|
||
**If you are an Airbyte core member**, copy the credentials in RPass under the secret name `source-hubspot-integration-test-config` | ||
and place them into `secrets/config.json`. | ||
|
||
|
||
### Locally running the connector | ||
``` | ||
python main_dev.py spec | ||
python main_dev.py check --config secrets/config.json | ||
python main_dev.py discover --config secrets/config.json | ||
python main_dev.py read --config secrets/config.json --catalog sample_files/configured_catalog.json | ||
``` | ||
|
||
### Unit Tests | ||
To run unit tests locally, from the connector directory run: | ||
``` | ||
pytest unit_tests | ||
``` | ||
|
||
### Locally running the connector docker image | ||
``` | ||
# in airbyte root directory | ||
./gradlew :airbyte-integrations:connectors:source-hubspot:airbyteDocker | ||
docker run --rm airbyte/source-hubspot:dev spec | ||
docker run --rm -v $(pwd)/airbyte-integrations/connectors/source-hubspot/secrets:/secrets airbyte/source-hubspot:dev check --config /secrets/config.json | ||
docker run --rm -v $(pwd)/airbyte-integrations/connectors/source-hubspot/secrets:/secrets airbyte/source-hubspot:dev discover --config /secrets/config.json | ||
docker run --rm -v $(pwd)/airbyte-integrations/connectors/source-hubspot/secrets:/secrets -v $(pwd)/airbyte-integrations/connectors/source-hubspot/sample_files:/sample_files airbyte/source-hubspot:dev read --config /secrets/config.json --catalog /sample_files/configured_catalog.json | ||
``` | ||
|
||
### Integration Tests | ||
1. From the airbyte project root, run `./gradlew :airbyte-integrations:connectors:source-hubspot:standardSourceTestPython` to run the standard integration test suite. | ||
1. To run additional integration tests, place your integration tests in a new directory `integration_tests` and run them with `pytest integration_tests`. | ||
Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named. | ||
|
||
## Dependency Management | ||
All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development. |
32 changes: 32 additions & 0 deletions
32
airbyte-integrations/connectors/source-hubspot/build.gradle
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
plugins { | ||
id 'airbyte-python' | ||
id 'airbyte-docker' | ||
id 'airbyte-standard-source-test-file' | ||
} | ||
|
||
airbytePython { | ||
moduleDirectory 'source_hubspot' | ||
} | ||
|
||
airbyteStandardSourceTestFile { | ||
// For more information on standard source tests, see https://docs.airbyte.io/contributing-to-airbyte/building-new-connector/testing-connectors | ||
|
||
// All these input paths must live inside this connector's directory (or subdirectories) | ||
// TODO update the spec JSON file | ||
specPath = "source_hubspot/spec.json" | ||
|
||
// configPath points to a config file which matches the spec.json supplied above. secrets/ is gitignored by default, so place your config file | ||
// there (in case it contains any credentials) | ||
// TODO update the config file to contain actual credentials | ||
configPath = "secrets/config.json" | ||
// TODO update the sample configured_catalog JSON for use in testing | ||
// Note: If your source supports incremental syncing, then make sure that the catalog that is returned in the get_catalog method is configured | ||
// for incremental syncing (e.g. include cursor fields, etc). | ||
configuredCatalogPath = "sample_files/configured_catalog.json" | ||
} | ||
|
||
|
||
dependencies { | ||
implementation files(project(':airbyte-integrations:bases:base-standard-source-test-file').airbyteDocker.outputs) | ||
implementation files(project(':airbyte-integrations:bases:base-python').airbyteDocker.outputs) | ||
} |
32 changes: 32 additions & 0 deletions
32
airbyte-integrations/connectors/source-hubspot/main_dev.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
""" | ||
MIT License | ||
|
||
Copyright (c) 2020 Airbyte | ||
|
||
Permission is hereby granted, free of charge, to any person obtaining a copy | ||
of this software and associated documentation files (the "Software"), to deal | ||
in the Software without restriction, including without limitation the rights | ||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
copies of the Software, and to permit persons to whom the Software is | ||
furnished to do so, subject to the following conditions: | ||
|
||
The above copyright notice and this permission notice shall be included in all | ||
copies or substantial portions of the Software. | ||
|
||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
SOFTWARE. | ||
""" | ||
|
||
import sys | ||
|
||
from base_python.entrypoint import launch | ||
from source_hubspot import SourceHubspot | ||
|
||
if __name__ == "__main__": | ||
source = SourceHubspot() | ||
launch(source, sys.argv[1:]) |
4 changes: 4 additions & 0 deletions
4
airbyte-integrations/connectors/source-hubspot/requirements.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
-e ../../bases/airbyte-protocol | ||
-e ../../bases/base-python | ||
-e ../../bases/base-python-test | ||
-e . |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should only make the new version available once incremental is complete. We can then also completely remove the old hubspot connector (rather than rename it).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
my intention was to add a completely new connector, then release it and drop the one that based on singer. But maybe accidentally renamed the old instead, not sure, need to check
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Important thing is not to have 2 hubspot connectors in Airbyte at the same time. the user only cares about syncing data from hubspot, Singer vs. native is an implementation detail
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
clear, I have removed the old connector