Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates repo to match changes to template #25

Merged
merged 5 commits into from
Sep 8, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .cruft.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"template": "https://github.com/PrefectHQ/prefect-collection-template",
"commit": "8cbce0373595b095876948479aacae3ac1deccb9",
"commit": "df6ab40ef55c4e1ff9cfadbecd0c6ee2da9dc560",
"checkout": null,
"context": {
"cookiecutter": {
Expand Down
24 changes: 24 additions & 0 deletions .github/workflows/publish-docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
name: Publish docs

on:
workflow_dispatch

jobs:
build-and-publish-docs:
name: Build and publish docs
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3

- name: Build docs
run: |
python -m pip install --upgrade pip
python -m pip install --upgrade --upgrade-strategy eager -e .[dev]
mkdocs build

- name: Publish docs
uses: JamesIves/github-pages-deploy-action@v4.4.0
with:
branch: docs
folder: site
61 changes: 61 additions & 0 deletions .github/workflows/template-sync.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
name: Template Synchronization
on:
schedule:
- cron: "0 0 * * *"
workflow_dispatch:

jobs:
check-for-update:
name: Check for template update
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: 3.9

- name: Install cruft
run: |
pip install "cookiecutter>=1.7.3,<2.0.0" cruft


- name: Check for update'
id: check
run: |
cruft check

submit-update-pr:
name: Submit update PR
runs-on: ubuntu-latest
needs: [check-for-update]
if: failure()
steps:
- uses: actions/checkout@v3

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: 3.9

- name: Install cruft
run: pip install cruft

- name: Perform updates
run: cruft update -y

- name: Submit PR
uses: peter-evans/create-pull-request@v4
with:
commit-message: Updating collection with changes to prefect-collection-template
branch: sync-with-template
delete-branch: true
title: Sync Collection with changes to prefect-collection-template
body: |
Automated PR created to propagate changes from prefect-collection-template to this collection

Feel free to make any necessary changes to this PR before merging.
labels: |
template sync
automated pr
20 changes: 0 additions & 20 deletions CONTRIBUTORS.md

This file was deleted.

60 changes: 56 additions & 4 deletions MAINTAINERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,40 @@ Requires an installation of Python 3.7+

We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.

### GitHub setup

Generate a Prefect Collection project in the terminal:

```bash
cookiecutter https://github.com/PrefectHQ/prefect-collection-template
```

Create a Git respoitory for the newly generated collection and create the first commit:

```bash
git init
git add .
git commit -m "Initial commit: project generated by prefect-collection-template"
```

Then, create a new repo following the prompts at:
https://github.com/organizations/PrefectHQ/repositories/new

Upon creation, push the repository to GitHub:
```bash
git remote add origin https://github.com/PrefectHQ/prefect-airbyte.git
git branch -M main
git push -u origin main
```

It's recommended to setup some protection rules for main at:
https://github.com/PrefectHQ/prefect-airbyte/settings/branches

- Require a pull request before merging
- Require approvals

Lastly, [code owners](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners) for the repository can be set, like this [example here](https://github.com/PrefectHQ/prefect/blob/master/.github/CODEOWNERS).

### Project setup

To setup your project run the following:
Expand All @@ -21,7 +55,7 @@ To setup your project run the following:
pre-commit install
```

To verify the set up was successful you can run the following:
To verify the setup was successful you can run the following:
- Run the tests for the example tasks and flow in the bootstrapped collection
```bash
pytest tests
Expand All @@ -41,21 +75,39 @@ For information about the use and development of tasks and flow, check out the [

This collection has been setup to with [mkdocs](https://www.mkdocs.org/) for automatically generated documentation. The signatures and docstrings of your tasks and flow will be used to generate documentation for the users of this collection. You can make changes to the structure of the generated documentation by editing the `mkdocs.yml` file in this project.

To add a new page for a module in your collection, create a new markdown file in the `docs` directory and add that file to the `nav` section of `mkdocs.yml`. If you want to automatically generate documentation based on the docstrings and signatures of the contents of the module with `mkdocstrings`, add a line to the new markdown file in the following format:

```markdown
::: prefect_airbyte.{module_name}
```

You can also refer to the `flows.md` and `tasks.md` files included in your generated project as examples.

Once you have working code, replace the default "Write and run a flow" example in `README.md` to match your collection.

## Development lifecycle

### CI Pipeline

This collection comes with [GitHub Actions](https://docs.github.com/en/actions) for testing and linting. To add additional actions, you can add jobs in the `.github/workflows` folder. On PR, the pipeline will run linting via [`black`](https://black.readthedocs.io/en/stable/) and [`flake8`](https://flake8.pycqa.org/en/latest/) and unit tests via `pytest`.
This collection comes with [GitHub Actions](https://docs.github.com/en/actions) for testing and linting. To add additional actions, you can add jobs in the `.github/workflows` folder. Upon a pull request, the pipeline will run linting via [`black`](https://black.readthedocs.io/en/stable/), [`flake8`](https://flake8.pycqa.org/en/latest/), [`interrogate`](https://interrogate.readthedocs.io/en/latest/), and unit tests via `pytest` alongside `coverage`.

`interrogate` will tell you which methods, functions, classes, and modules have docstrings, and which do not--the job has a fail threshold of 95%, meaning that it will fail if more than 5% of the codebase is undocumented. We recommend following the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings) for docstring format.

Simiarly, `coverage` ensures that the codebase includes tests--the job has a fail threshold of 80%, meaning that it will fail if more than 20% of the codebase is missing tests.

### Package and Publish

GitHub actions will also handle packaging and publishing of your collection to [PyPI](https://pypi.org/) so other Prefect users can your collection in their flows.

In order to publish to PyPI, you'll need a PyPI account and generate an API token to authenticate with PyPI when publishing new versions of your collection. The [PyPI documentation](https://pypi.org/help/#apitoken) outlines the steps needed to get an API token.
To publish to PyPI, you'll need a PyPI account and to generate an API token to authenticate with PyPI when publishing new versions of your collection. The [PyPI documentation](https://pypi.org/help/#apitoken) outlines the steps needed to get an API token.

Once you've obtained a PyPI API token, [create a GitHub secret](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository) named `PYPI_API_TOKEN`.

There is also a GitHub Action that performs a test publish to [test PyPI](https://test.pypi.org/) to allow for a test deployment without affecting PyPI. You will need a separate token for test PyPI which can be saved in a GitHub secret names `TEST_PYPI_API_TOKEN`
To publish a new version of your collection, [create a new GitHub release](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository#creating-a-release) and tag it with the version that you want to deploy (e.g. v0.3.2). This will trigger a workflow to publish the new version on PyPI and deploy the updated docs to GitHub pages.

Upon publishing, a `docs` branch is automatically created. To hook this up to GitHub Pages, simply head over to https://github.com/PrefectHQ/prefect-airbyte/settings/pages, select `docs` under the dropdown menu, keep the default `/root` folder, `Save`, and upon refresh, you should see a prompt stating "Your site is published at https://PrefectHQ.github.io/prefect-airbyte". Don't forget to add this link to the repo's "About" section, under "Website" so users can access the docs easily.

Feel free to [submit your collection](https://orion-docs.prefect.io/collections/overview/#listing-in-the-collections-catalog) to the Prefect [Collections Catalog](https://orion-docs.prefect.io/collections/catalog/)!

## Further guidance

Expand Down
18 changes: 16 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,22 @@
# prefect-airbyte

<p align="center">
<a href="https://pypi.python.org/pypi/prefect-airbyte/" alt="PyPI version">
<img alt="PyPI" src="https://img.shields.io/pypi/v/prefect-airbyte?color=0052FF&labelColor=090422"></a>
<a href="https://github.com/PrefectHQ/prefect-airbyte/" alt="Stars">
<img src="https://img.shields.io/github/stars/PrefectHQ/prefect-airbyte?color=0052FF&labelColor=090422" /></a>
<a href="https://pepy.tech/badge/prefect-airbyte/" alt="Downloads">
<img src="https://img.shields.io/pypi/dm/prefect-airbyte?color=0052FF&labelColor=090422" /></a>
<a href="https://github.com/PrefectHQ/prefect-airbyte/pulse" alt="Activity">
<img src="https://img.shields.io/github/commit-activity/m/PrefectHQ/prefect-airbyte?color=0052FF&labelColor=090422" /></a>
<br>
<a href="https://prefect-airbyte-community.slack.com" alt="Slack">
<img src="https://img.shields.io/badge/slack-join_community-red.svg?color=0052FF&labelColor=090422&logo=slack" /></a>
<a href="https://discourse.prefect-airbyte.io/" alt="Discourse">
<img src="https://img.shields.io/badge/discourse-browse_forum-red.svg?color=0052FF&labelColor=090422&logo=discourse" /></a>
</p>

## Welcome!
<!-- &emsp; <img src="imgs/airbyte.png" width="40" height="55" /> -->

`prefect-airbyte` is a collection of prebuilt Prefect tasks that can be used to quickly construct Prefect flows to trigger Airbyte syncs or export your connector configurations.

Expand All @@ -26,7 +41,6 @@ pip install prefect-airbyte
### Airbyte setup
See [the airbyte documention](https://docs.airbyte.com/deploying-airbyte) on how to get your own instance.


### Examples

#### Trigger a defined connection sync
Expand Down
47 changes: 47 additions & 0 deletions docs/stylesheets/extra.css
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,50 @@ to force column width */
white-space: nowrap;
}

/* badge formatting */
.badge::before {
background-color: #1860F2;
color: white;
font-size: 0.8rem;
font-weight: normal;
padding: 4px 8px;
margin-left: 0.5rem;
vertical-align: super;
text-align: center;
border-radius: 5px;
}

.badge-api::before {
background-color: #1860F2;
color: white;
font-size: 0.8rem;
font-weight: normal;
padding: 4px 8px;
text-align: center;
border-radius: 5px;
}

.experimental::before {
background-color: #FCD14E;
content: "Experimental";
}

.cloud::before {
background-color: #799AF7;
content: "Prefect Cloud";
}

.deprecated::before {
background-color: #FA1C2F;
content: "Deprecated";
}

.new::before {
background-color: #2AC769;
content: "New";
}

.expert::before {
background-color: #726576;
content: "Advanced";
}
7 changes: 4 additions & 3 deletions mkdocs.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
site_name: prefect-airbyte
site_url: https://prefecthq.github.io/prefect-airbyte
repo_url: https://github.com/prefecthq/prefect-airbyte
site_url: https://PrefectHQ.github.io/prefect-airbyte
repo_url: https://github.com/PrefectHQ/prefect-airbyte
edit_uri: /edit/main/docs/
theme:
name: material
Expand All @@ -10,7 +10,8 @@ theme:
accent: blue
icon:
repo: fontawesome/brands/github
logo: img/prefect-logo-white.png
logo:
img/prefect-logo-white.png
font:
text: Inter
code: Source Code Pro
Expand Down
2 changes: 1 addition & 1 deletion requirements-dev.txt
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,4 @@ mock; python_version < '3.8'
mkdocs-gen-files
interrogate
coverage
respx
respx
2 changes: 1 addition & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
prefect>=2.0b2
prefect>=2.0.0
2 changes: 2 additions & 0 deletions setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ parentdir_prefix =

[tool:interrogate]
ignore-init-module = True
ignore_init_method = True
exclude = prefect_airbyte/_version.py, tests, setup.py, versioneer.py, docs, site
fail-under = 95
omit-covered-files = True
Expand All @@ -32,6 +33,7 @@ omit = tests/*, prefect_airbyte/_version.py

[coverage:report]
fail_under = 80
show_missing = True

[tool:pytest]
asyncio_mode = auto