-
Notifications
You must be signed in to change notification settings - Fork 3.8k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[connector-templates] Switch python destination template to generate …
…pyproject.toml instead of setup.py and requirements.txt (#37457) Co-authored-by: Natik Gadzhi <natik@respawn.io>
- Loading branch information
1 parent
383af5c
commit 614e15f
Showing
7 changed files
with
147 additions
and
185 deletions.
There are no files selected for viewing
159 changes: 0 additions & 159 deletions
159
airbyte-integrations/connector-templates/destination-python/README.md
This file was deleted.
Oops, something went wrong.
103 changes: 103 additions & 0 deletions
103
airbyte-integrations/connector-templates/destination-python/README.md.hbs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,103 @@ | ||
# {{capitalCase name}} Destination | ||
|
||
This is the repository for the {{capitalCase name}} destination connector, written in Python. | ||
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/destinations/{{dashCase name}}). | ||
|
||
## Local development | ||
|
||
### Prerequisites | ||
|
||
* Python (`^3.9`) | ||
* Poetry (`^1.7`) - installation instructions [here](https://python-poetry.org/docs/#installation) | ||
|
||
|
||
|
||
### Installing the connector | ||
|
||
From this connector directory, run: | ||
```bash | ||
poetry install --with dev | ||
``` | ||
|
||
|
||
#### Create credentials | ||
|
||
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/destinations/{{dashCase name}}) | ||
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `destination_{{snakeCase name}}/spec.json` file. | ||
Note that the `secrets` directory is gitignored by default, so there is no danger of accidentally checking in sensitive information. | ||
See `integration_tests/sample_config.json` for a sample config file. | ||
|
||
**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `destination {{dashCase name}} test creds` | ||
and place them into `secrets/config.json`. | ||
|
||
### Locally running the connector | ||
``` | ||
poetry run destination-{{dashCase name}} spec | ||
poetry run destination-{{dashCase name}} check --config secrets/config.json | ||
poetry run destination-{{dashCase name}} write --config secrets/config.json --catalog sample_files/configured_catalog.json | ||
``` | ||
|
||
### Running tests | ||
|
||
To run tests locally, from the connector directory run: | ||
|
||
``` | ||
poetry run pytest tests | ||
``` | ||
|
||
### Building the docker image | ||
|
||
1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) | ||
2. Run the following command to build the docker image: | ||
```bash | ||
airbyte-ci connectors --name=destination-{{dashCase name}} build | ||
``` | ||
|
||
An image will be available on your host with the tag `airbyte/destination-{{dashCase name}}:dev`. | ||
|
||
### Running as a docker container | ||
|
||
Then run any of the connector commands as follows: | ||
``` | ||
docker run --rm airbyte/destination-{{dashCase name}}:dev spec | ||
docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-{{dashCase name}}:dev check --config /secrets/config.json | ||
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/destination-{{dashCase name}}:dev write --config /secrets/config.json --catalog /integration_tests/configured_catalog.json | ||
``` | ||
|
||
### Running our CI test suite | ||
|
||
You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md): | ||
|
||
```bash | ||
airbyte-ci connectors --name=destination-{{dashCase name}} test | ||
``` | ||
|
||
### Customizing acceptance Tests | ||
|
||
Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information. | ||
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py. | ||
|
||
### Dependency Management | ||
|
||
All of your dependencies should be managed via Poetry. | ||
To add a new dependency, run: | ||
|
||
```bash | ||
poetry add <package-name> | ||
``` | ||
|
||
Please commit the changes to `pyproject.toml` and `poetry.lock` files. | ||
|
||
## Publishing a new version of the connector | ||
|
||
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what? | ||
1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=destination-{{dashCase name}} test` | ||
2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)): | ||
- bump the `dockerImageTag` value in in `metadata.yaml` | ||
- bump the `version` value in `pyproject.toml` | ||
3. Make sure the `metadata.yaml` content is up to date. | ||
4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/destinations/{{dashCase name}}.md`). | ||
5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention). | ||
6. Pat yourself on the back for being an awesome contributor. | ||
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master. | ||
8. Once your PR is merged, the new version of the connector will be automatically published to Docker Hub and our connector registry. |
13 changes: 13 additions & 0 deletions
13
...grations/connector-templates/destination-python/destination_{{snakeCase name}}/run.py.hbs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
# | ||
# Copyright (c) {{currentYear}} Airbyte, Inc., all rights reserved. | ||
# | ||
|
||
|
||
import sys | ||
|
||
from airbyte_cdk.entrypoint import launch | ||
from .destination import Destination{{properCase name}} | ||
|
||
def run(): | ||
destination = Destination{{properCase name}}() | ||
destination.run(sys.argv[1:]) |
28 changes: 28 additions & 0 deletions
28
airbyte-integrations/connector-templates/destination-python/pyproject.toml.hbs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
[build-system] | ||
requires = [ "poetry-core>=1.0.0",] | ||
build-backend = "poetry.core.masonry.api" | ||
|
||
[tool.poetry] | ||
version = "0.1.0" | ||
name = "destination-{{dashCase name}}" | ||
description = "Destination implementation for {{dashCase name}}." | ||
authors = [ "Airbyte <contact@airbyte.io>",] | ||
license = "MIT" | ||
readme = "README.md" | ||
documentation = "https://docs.airbyte.com/integrations/destinations/{{dashCase name}}" | ||
homepage = "https://airbyte.com" | ||
repository = "https://github.com/airbytehq/airbyte" | ||
packages = [ { include = "destination_{{snakeCase name}}" }, {include = "main.py" } ] | ||
|
||
[tool.poetry.dependencies] | ||
python = "^3.9,<3.12" | ||
airbyte-cdk = "^0" | ||
[tool.poetry.scripts] | ||
destination-{{dashCase name}} = "destination_{{snakeCase name}}.run:run" | ||
[tool.poetry.group.dev.dependencies] | ||
requests-mock = "*" | ||
pytest-mock = "*" | ||
pytest = "*" | ||
1 change: 0 additions & 1 deletion
1
airbyte-integrations/connector-templates/destination-python/requirements.txt
This file was deleted.
Oops, something went wrong.
3 changes: 3 additions & 0 deletions
3
airbyte-integrations/connector-templates/destination-python/secrets/config.json.hbs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
{ | ||
"fix-me": "TODO populate with needed configuration for integration tests or delete this file and any references to it. The schema of this file should match what is in your spec.yaml" | ||
} |
25 changes: 0 additions & 25 deletions
25
airbyte-integrations/connector-templates/destination-python/setup.py.hbs
This file was deleted.
Oops, something went wrong.