Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Do not merge] - test workflows without elevated perms #36367

Closed
wants to merge 10 commits into from
6 changes: 5 additions & 1 deletion .github/workflows/contractors_review_requirements.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,11 @@ jobs:
name: "Check if a review is required from Connector teams"
runs-on: ubuntu-latest

if: ${{ github.repository == 'airbytehq/airbyte' }}
if: ${{ github.event.repository.fork == false }}
# This workflow cannot run on forks, as the fork's github token does not have `read:org`
# permissions, which are required to check the user's team membership. We assume that a
# review on a fork's PR is always required from one or more connector teams and/or support.

steps:
- name: Check contributor team membership
uses: tspascoal/get-user-teams-membership@v3
Expand Down
9 changes: 0 additions & 9 deletions .github/workflows/format_fix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,21 +17,12 @@ jobs:
uses: actions/checkout@v3
with:
ref: ${{ github.ref }}
# Important that this is set so that CI checks are triggered again
# Without this we would be forever waiting on required checks to pass
token: ${{ secrets.GH_PAT_APPROVINGTON_OCTAVIA }}

- name: Run airbyte-ci format fix all
uses: ./.github/actions/run-airbyte-ci
continue-on-error: true
with:
context: "manual"
dagger_cloud_token: ${{ secrets.DAGGER_CLOUD_TOKEN_2 }}
docker_hub_password: ${{ secrets.DOCKER_HUB_PASSWORD }}
docker_hub_username: ${{ secrets.DOCKER_HUB_USERNAME }}
gcs_credentials: ${{ secrets.METADATA_SERVICE_PROD_GCS_CREDENTIALS }}
sentry_dsn: ${{ secrets.SENTRY_AIRBYTE_CI_DSN }}
github_token: ${{ secrets.GH_PAT_MAINTENANCE_OCTAVIA }}
subcommand: "format fix all"

# This is helpful in the case that we change a previously committed generated file to be ignored by git.
Expand Down
3 changes: 2 additions & 1 deletion airbyte-integrations/connectors/destination-duckdb/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,10 @@ cat integration_tests/messages.jsonl| python main.py write --config integration_
#### Build
**Via [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) (recommended):**
```bash
airbyte-ci connectors --name=destination-duckdb build
airbyte-ci connectors --name=destination-duckdb build [--architecture=...]
```


An image will be built with the tag `airbyte/destination-duckdb:dev`.

**Via `docker build`:**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,10 +81,12 @@ def write(

# Get and register auth token if applicable
motherduck_api_key = str(config.get(CONFIG_MOTHERDUCK_API_KEY, ""))
duckdb_config = {}
if motherduck_api_key:
os.environ["motherduck_token"] = motherduck_api_key
duckdb_config["motherduck_token"] = motherduck_api_key
duckdb_config["custom_user_agent"] = "airbyte"

con = duckdb.connect(database=path, read_only=False)
con = duckdb.connect(database=path, read_only=False, config=duckdb_config)

con.execute(f"CREATE SCHEMA IF NOT EXISTS {schema_name}")

Expand Down Expand Up @@ -175,10 +177,13 @@ def check(self, logger: AirbyteLogger, config: Mapping[str, Any]) -> AirbyteConn
logger.info(f"Using DuckDB file at {path}")
os.makedirs(os.path.dirname(path), exist_ok=True)

duckdb_config = {}
if CONFIG_MOTHERDUCK_API_KEY in config:
os.environ["motherduck_token"] = str(config[CONFIG_MOTHERDUCK_API_KEY])
duckdb_config["motherduck_token"] = str(config[CONFIG_MOTHERDUCK_API_KEY])
duckdb_config["custom_user_agent"] = "airbyte"

con = duckdb.connect(database=path, read_only=False)

con = duckdb.connect(database=path, read_only=False, config=duckdb_config)
con.execute("SELECT 1;")

return AirbyteConnectionStatus(status=Status.SUCCEEDED)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
Type,
)
from destination_duckdb import DestinationDuckdb
from destination_duckdb.destination import CONFIG_MOTHERDUCK_API_KEY

CONFIG_PATH = "integration_tests/config.json"
SECRETS_CONFIG_PATH = "secrets/config.json" # Should contain a valid MotherDuck API token
Expand Down Expand Up @@ -179,8 +180,12 @@ def test_write(

result = list(generator)
assert len(result) == 1

con = duckdb.connect(database=config.get("destination_path"), read_only=False)
motherduck_api_key = str(config.get(CONFIG_MOTHERDUCK_API_KEY, ""))
duckdb_config = {}
if motherduck_api_key:
duckdb_config["motherduck_token"] = motherduck_api_key
duckdb_config["custom_user_agent"] = "airbyte_intg_test"
con = duckdb.connect(database=config.get("destination_path"),read_only=False, config=duckdb_config)
with con:
cursor = con.execute(
"SELECT _airbyte_ab_id, _airbyte_emitted_at, _airbyte_data "
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ data:
connectorSubtype: database
connectorType: destination
definitionId: 94bd199c-2ff0-4aa2-b98e-17f0acb72610
dockerImageTag: 0.3.1
dockerImageTag: 0.3.2
dockerRepository: airbyte/destination-duckdb
githubIssueLabel: destination-duckdb
icon: duckdb.svg
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[tool.poetry]
name = "destination-duckdb"
version = "0.3.0"
version = "0.3.2"
description = "Destination implementation for Duckdb."
authors = ["Simon Späti, Airbyte"]
license = "MIT"
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/duckdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,7 @@ Note: If you are running Airbyte on Windows with Docker backed by WSL2, you have

| Version | Date | Pull Request | Subject |
| :------ | :--------- | :------------------------------------------------------- | :--------------------- |
| 0.3.2 | 2024-03-20 | [#32635](https://github.com/airbytehq/airbyte/pull/32635) | Instrument custom_user_agent to identify Airbyte-Motherduck connector usage. |
| 0.3.1 | 2023-11-18 | [#32635](https://github.com/airbytehq/airbyte/pull/32635) | Upgrade DuckDB version to [`v0.9.2`](https://github.com/duckdb/duckdb/releases/tag/v0.9.2). |
| 0.3.0 | 2022-10-23 | [#31744](https://github.com/airbytehq/airbyte/pull/31744) | Upgrade DuckDB version to [`v0.9.1`](https://github.com/duckdb/duckdb/releases/tag/v0.9.1). **Required update for all MotherDuck users.** Note, this is a **BREAKING CHANGE** for users who may have other connections using versions of DuckDB prior to 0.9.x. See the [0.9.0 release notes](https://github.com/duckdb/duckdb/releases/tag/v0.9.0) for more information and for upgrade instructions. |
| 0.2.1 | 2022-10-20 | [#30600](https://github.com/airbytehq/airbyte/pull/30600) | Fix: schema name mapping |
Expand Down