-
Notifications
You must be signed in to change notification settings - Fork 3.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Allow us to publish CDK versions on PyPi via Github workflows. This lets us: - Version CDK moving forward - Publish via Github actions to both PyPi and test PyPi for test releases. Note, this will not work on this branch as Github only detects new workflows after they are checked into master, so I was forced to develop and test this from a fork. Browse this PR or this PR to see this command in action. Added publishing instructions to the Release Document.
- Loading branch information
Showing
12 changed files
with
189 additions
and
22 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
name: Publish CDK | ||
on: | ||
workflow_dispatch: | ||
inputs: | ||
dry-run: | ||
description: 'Indicates whether this is a dry-run or not. A dry-run publishes to Test PyPi. A proper run publishes to actual PyPi servers.' | ||
required: false | ||
comment-id: | ||
description: 'The comment-id of the slash command. Used to update the comment with the status.' | ||
required: false | ||
|
||
jobs: | ||
publish-cdk: | ||
runs-on: ubuntu-latest | ||
steps: | ||
- name: Link comment to workflow run | ||
if: github.event.inputs.comment-id | ||
uses: peter-evans/create-or-update-comment@v1 | ||
with: | ||
comment-id: ${{ github.event.inputs.comment-id }} | ||
body: | | ||
> :clock2: ${{github.event.inputs.connector}} https://github.com/${{github.repository}}/actions/runs/${{github.run_id}} | ||
# Make use of env vars to dynamically set the PyPi url. Since the default is set to publish to production PyPi, only set the url if during | ||
# a dry-run publish to the Test PyPi servers. | ||
- name: Publish to test PyPi if dry-run. | ||
if: github.event.inputs.dry-run != 'false' | ||
run: | | ||
echo ${{ github.event.inputs.dry-run }} | ||
echo "pypi_url=https://test.pypi.org/legacy/" >> $GITHUB_ENV | ||
- name: Checkout Airbyte | ||
uses: actions/checkout@v2 | ||
- name: Publish Python Package | ||
uses: mariamrf/py-package-publish-action@v1.1.0 | ||
with: | ||
python_version: '3.7.9' | ||
pip_version: '21.1' | ||
subdir: 'airbyte-cdk/python/' | ||
env: | ||
TWINE_PASSWORD: ${{ secrets.TWINE_PASSWORD }} | ||
TWINE_USERNAME: ${{ secrets.TWINE_USERNAME }} | ||
TWINE_REPOSITORY_URL: ${{ env.pypi_url }} | ||
- name: Add Success Comment | ||
if: github.event.inputs.comment-id && success() | ||
uses: peter-evans/create-or-update-comment@v1 | ||
with: | ||
comment-id: ${{ github.event.inputs.comment-id }} | ||
body: | | ||
> :white_check_mark: ${{github.event.inputs.connector}} https://github.com/${{github.repository}}/actions/runs/${{github.run_id}} | ||
- name: Add Failure Comment | ||
if: github.event.inputs.comment-id && !success() | ||
uses: peter-evans/create-or-update-comment@v1 | ||
with: | ||
comment-id: ${{ github.event.inputs.comment-id }} | ||
body: | | ||
> :x: ${{github.event.inputs.connector}} https://github.com/${{github.repository}}/actions/runs/${{github.run_id}} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
Copyright (c) 2020 Airbyte | ||
|
||
Permission is hereby granted, free of charge, to any person obtaining a copy | ||
of this software and associated documentation files (the "Software"), to deal | ||
in the Software without restriction, including without limitation the rights | ||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
copies of the Software, and to permit persons to whom the Software is | ||
furnished to do so, subject to the following conditions: | ||
|
||
The above copyright notice and this permission notice shall be included in all | ||
copies or substantial portions of the Software. | ||
|
||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
SOFTWARE. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
# Airbyte Python Connector Development Framework (CDK) | ||
|
||
The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors. | ||
The CDK currently offers helpers specific for creating Airbyte source connectors for: | ||
* HTTP APIs (REST APIs, GraphQL, etc..) | ||
* Singer Taps | ||
* Generic Python sources (anything not covered by the above) | ||
|
||
The CDK provides an improved developer experience by providing basic implementation structure and abstracting away low-level glue boilerplate. | ||
|
||
This document is a general introduction to the CDK. Readers should have basic familiarity with the [Airbyte Specification](https://docs.airbyte.io/architecture/airbyte-specification) before proceeding. | ||
|
||
## Getting started | ||
Generate an empty connector using the code generator. First clone the Airbyte repository then from the repository root run | ||
``` | ||
cd airbyte-integrations/connector-templates/generator | ||
npm run generate | ||
``` | ||
|
||
then follow the interactive prompt. Next, find all `TODO`s in the generated project directory -- they're accompanied by lots of comments explaining what you'll need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector. | ||
|
||
Additionally, you can follow [this tutorial](docs/tutorials/http_api_source.md) for a complete walkthrough of creating an HTTP connector using the Airbyte CDK. | ||
|
||
### Concepts & Documentation | ||
See the [overview docs](docs/concepts/overview.md) for a tour through what the API offers. | ||
|
||
### Airbyte Specification | ||
Find the reference docs for the Airbyte Specification (the interface for how sources and destinations interact) [here](https://docs.airbyte.io/architecture/airbyte-specification). | ||
|
||
### Example Connectors | ||
|
||
**HTTP Connectors**: | ||
* [Exchangerates API](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-exchange-rates/source_exchange_rates/source.py) | ||
* [Stripe](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-stripe/source_stripe/source.py) | ||
* [Slack](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-slack/source_slack/source.py) | ||
|
||
**Singer connectors**: | ||
* [Salesforce](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-salesforce-singer/source_salesforce_singer/source.py) | ||
* [Github](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github-singer/source_github_singer/source.py) | ||
|
||
**Simple Python connectors using the barebones `Source` abstraction**: | ||
* [Google Sheets](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-google-sheets/google_sheets_source/google_sheets_source.py) | ||
* [Mailchimp](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mailchimp/source_mailchimp/source.py) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
# Defines Python build system settings. | ||
[build-system] | ||
requires = [ | ||
"setuptools>=42", | ||
"wheel" | ||
] | ||
|
||
build-backend = "setuptools.build_meta" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3 changes: 3 additions & 0 deletions
3
airbyte-integrations/connectors/source-exchange-rates/sample_files/state.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
{ | ||
"exchange_rates": {"date": "2021-04-28"} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters