✨ Destination DuckDB: Use latest DuckDB 0.9.1 libraries 🚨(breaking change)🚨

[aaronsteers]

What

Use the latest DuckDB version as of today, which is v0.9.1.

This re-enables connectivity to MotherDuck, which now uses DuckDB 0.9.x.

How

Simple bump of the DuckDB libraries. No code changes.

Recommended reading order

1. First read the contributing docs, which document this process.
2. (Everything else.)

🚨 User Impact 🚨

The v0.9.x DuckDB database format is not backwards compatible with versions prior to 0.9.0.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Oct 24, 2023 9:45pm

[github-actions]

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

• PR name follows PR naming conventions
• Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
• Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
• You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
• Secrets in the connector's spec are annotated with airbyte_secret
• All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
• Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
• Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
• If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

1. Check for hidden checklists in your PR description

2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

[airbyte-oss-build-runner]

destination-duckdb test report (commit 556c4ae10d) - ❌

⏲️ Total pipeline duration: 33.79s

Step	Result
Build destination-duckdb docker image for platform(s) linux/amd64	❌
Code format checks	✅
Validate metadata for destination-duckdb	❌
Connector version semver check	✅
Connector version increment check	❌
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-duckdb test

[airbyte-oss-build-runner]

destination-duckdb test report (commit a382ec1fca) - ❌

⏲️ Total pipeline duration: 20.74s

Step	Result
Build destination-duckdb docker image for platform(s) linux/amd64	❌
Code format checks	✅
Validate metadata for destination-duckdb	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-duckdb test

[airbyte-oss-build-runner]

destination-duckdb test report (commit ac1214f2c5) - ❌

⏲️ Total pipeline duration: 6.53s

Step	Result
Build destination-duckdb docker image for platform(s) linux/amd64	❌
Code format checks	✅
Validate metadata for destination-duckdb	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-duckdb test

[airbyte-oss-build-runner]

destination-duckdb test report (commit 0ce82e4fa3) - ❌

⏲️ Total pipeline duration: 21.80s

Step	Result
Build destination-duckdb docker image for platform(s) linux/amd64	❌
Code format checks	✅
Validate metadata for destination-duckdb	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-duckdb test

[aaronsteers]

@alafanechere, @bnchrch - I've copied over the base image config from:

• Migrate connectors to use our python base image (Round 1) #31543

Lmk if this looks okay.

[alafanechere]

The migration to the base image looks good to me! Thank you.
Please write a migration guide in docs/integrations/destinations/duckdb-migrations.md to document how users should upgrade DuckDb.

[alafanechere]

Suggested change

	dockerImageTag: 0.3.0
	dockerImageTag: 1.0.0

It's a breaking change, can we bump by a major version?

[aaronsteers]

We spoke offline. This is community connector and doesn't receive official support.

Also, DuckDB's own database version files are incompatible across minor versions, and the best way to respect this volatility is to keep pre-1.0 connector status until we have a handle on stability overall.

Last - there's a desire to rewrite this in Java before we give official/certified status to the connector.

[bnchrch]

Looks like the only issue here is making it a major version change instead of a minor

[bnchrch]

Suggested change

	0.3.0:
	1.0.0:

Just adding the major version suggestion here as well

[aaronsteers]

Hi, @bnchrch

https://github.com/airbytehq/airbyte/pull/31744/files#r1370836343

From my response on Augustine's comment above:

We spoke offline. This is community connector and doesn't receive official support.

Also, DuckDB's own database version files are incompatible across minor versions, and the best way to respect this volatility is to keep pre-1.0 connector status until we have a handle on stability overall.

Last - there's a desire to rewrite this in Java before we give official/certified status to the connector.

[bnchrch]

Ah fair enough @aaronsteers !

I was going to dig my heels in on "no major change no breaking change" but after a discussion with @pedroslopez he pointed out weve been doing this same pattern of minor version changes can be breaking if the connector is v0

langchain is the example that came to mind
https://github.com/airbytehq/airbyte/blob/c92837c8d7e7f5f84f63547f3f98ed541fb4defb/airbyte-integrations/connectors/destination-langchain/metadata.yaml

and that seems to line up with the official interpretation of semver when the version is still 0.

TLDR: LGTM!

[airbyte-oss-build-runner]

destination-duckdb test report (commit 8b601e9fde) - ❌

⏲️ Total pipeline duration: 01mn41s

Step	Result
Build destination-duckdb docker image for platform(s) linux/amd64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-duckdb	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-duckdb test

[airbyte-oss-build-runner]

destination-duckdb test report (commit 3d7d6b6fb0) - ✅

⏲️ Total pipeline duration: 01mn15s

Step	Result
Build destination-duckdb docker image for platform(s) linux/amd64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-duckdb	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-duckdb test

[airbyte-oss-build-runner]

destination-duckdb test report (commit f1697c11d1) - ✅

⏲️ Total pipeline duration: 50.28s

Step	Result
Build destination-duckdb docker image for platform(s) linux/amd64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-duckdb	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-duckdb test

[airbyte-oss-build-runner]

destination-duckdb test report (commit 889bdcbef3) - ✅

⏲️ Total pipeline duration: 53.16s

Step	Result
Build destination-duckdb docker image for platform(s) linux/amd64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-duckdb	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-duckdb test

[airbyte-oss-build-runner]

destination-duckdb test report (commit eec1bc82ed) - ✅

⏲️ Total pipeline duration: 44.46s

Step	Result
Build destination-duckdb docker image for platform(s) linux/amd64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-duckdb	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-duckdb test

[airbyte-oss-build-runner]

destination-duckdb test report (commit e19842a6a4) - ✅

⏲️ Total pipeline duration: 32.52s

Step	Result
Build destination-duckdb docker image for platform(s) linux/amd64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-duckdb	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-duckdb test

[airbyte-oss-build-runner]

destination-duckdb test report (commit 2289de6b30) - ✅

⏲️ Total pipeline duration: 28.47s

Step	Result
Build destination-duckdb docker image for platform(s) linux/amd64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-duckdb	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-duckdb test

[airbyte-oss-build-runner]

destination-duckdb test report (commit d5c6acacfa) - ✅

⏲️ Total pipeline duration: 36.96s

Step	Result
Build destination-duckdb docker image for platform(s) linux/amd64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-duckdb	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-duckdb test

pre-1.0 status is desired due to continued volatility (connector not certified)