✨ [destination-DuckDB] use `pyarrow` batch insert to replace executemany #36715

hrl20 · 2024-04-01T05:58:09Z

What

Improve the DuckDB connector performance by replacing the .executemany calls.

How

DuckDB docs recommends against using .executemany for ingestion https://duckdb.org/docs/api/python/dbapi. Pyarrow is a convenient way to represent the buffered records to do bulk insert into duckdb.

User Impact

No functional changes. Better ingestion performance.

In the MotherDuck integration test:
5000 in batches of 1000: before: ~258s; after: ~1.8s
1M records in batches of 1000: before:(not measured); after: 70s

vercel · 2024-04-01T05:58:15Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Apr 16, 2024 4:19am

CLAassistant · 2024-04-01T05:58:20Z

All committers have signed the CLA.

aaronsteers

@hrl20 - This looks great, first of all. Thanks very much for contributing! The performance boost is very exciting. 🔥

One question inline about complex/messy schemas.

airbyte-integrations/connectors/destination-duckdb/destination_duckdb/destination.py

marcosmarxm · 2024-04-08T17:17:57Z

@aaronsteers and @hrl20 make sure all contributors had sign the CLA to merge this contribution.

aaronsteers

Hi, @hrl20! Thanks very much for this submission!

A few minor changes requested. Then, I think this is ready to ship.

Once you've replied and/or updated the PR, if you can re-request my review, I'll try to review and get merged without much additional delay.

Thanks!

airbyte-integrations/connectors/destination-duckdb/destination_duckdb/destination.py

docs/integrations/destinations/duckdb.md

airbyte-integrations/connectors/destination-duckdb/destination_duckdb/destination.py

aaronsteers

Approved. 🎉 🚀

We still need to make sure CI tests pass. I'll try to get that done tomorrow or Wed.
cc @marcosmarxm in case you have any cycles to contribute towards this. (Totally ok either way.)

github-actions · 2024-04-18T06:32:17Z

✅ Tests passed. [🔗][1]

[1]:

github-actions · 2024-04-18T07:07:15Z

PR test job started... Check job output.

github-actions · 2024-04-18T07:11:26Z

✅ Tests passed. 📎

github-actions · 2024-04-22T21:13:02Z

PR test job started... Check job output.

✅ Tests passed. 📎

github-actions · 2024-04-22T21:56:23Z

PR test job started... Check job output.

✅ Tests passed. 📎

github-actions · 2024-04-22T22:12:11Z

PR test job started... Check job output.

✅ Tests passed. 📎

aaronsteers · 2024-04-22T22:20:27Z

I've run this through the airbyte-ci tests and applied the format_fix manual CI workflow action.

All have passed so I'm proceeding to merge.

Connector should auto-publish after merge.

…t as replacement of `executemany` (#36715)

…t as replacement of `executemany` (airbytehq#36715)

octavia-squidington-iii added area/connectors Connector related issues community connectors/destination/duckdb labels Apr 1, 2024

hrl20 force-pushed the duckdb-batching branch 3 times, most recently from 31f7967 to ddda446 Compare April 2, 2024 05:17

hrl20 changed the title ~~use pyarrow to batch insert into duckdb~~ ✨ Destination DuckDB: use pyarrow batch insert to replace executemany Apr 2, 2024

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Apr 2, 2024

vercel bot deployed to Preview April 2, 2024 15:49 View deployment

hrl20 marked this pull request as ready for review April 2, 2024 15:57

aaronsteers self-requested a review April 2, 2024 17:41

aaronsteers reviewed Apr 2, 2024

View reviewed changes

airbyte-integrations/connectors/destination-duckdb/destination_duckdb/destination.py Outdated Show resolved Hide resolved

hrl20 requested a review from aaronsteers April 4, 2024 18:58

hrl20 commented Apr 4, 2024

View reviewed changes

airbyte-integrations/connectors/destination-duckdb/destination_duckdb/destination.py Show resolved Hide resolved

hrl20 force-pushed the duckdb-batching branch from ddeb1ec to 809aeec Compare April 8, 2024 18:40

marcosmarxm changed the title ~~✨ Destination DuckDB: use pyarrow batch insert to replace executemany~~ ✨ [destination-DuckDB] use pyarrow batch insert to replace executemany Apr 10, 2024

marcosmarxm assigned aaronsteers Apr 10, 2024

aaronsteers requested changes Apr 15, 2024

View reviewed changes

hrl20 added 6 commits April 15, 2024 17:28

duckdb use pyarrow to batch insert into duckdb

67d82b1

large test

9fa8e4b

docs and manifest update

abe34ee

add some tests for json schema, and add fallback path.

ca77b3b

add some tests for json schema, and add fallback path.

21460de

address comments

061cdd5

hrl20 force-pushed the duckdb-batching branch from 809aeec to 061cdd5 Compare April 15, 2024 21:30

hrl20 requested a review from aaronsteers April 15, 2024 21:32

vercel bot deployed to Preview April 15, 2024 21:35 View deployment

aaronsteers reviewed Apr 15, 2024

View reviewed changes

airbyte-integrations/connectors/destination-duckdb/destination_duckdb/destination.py Outdated Show resolved Hide resolved

fix typo

c96512c

aaronsteers approved these changes Apr 15, 2024

View reviewed changes

update poetry.lock

6a43221

This comment was marked as outdated.

Sign in to view

aaronsteers merged commit d4944a3 into airbytehq:master Apr 22, 2024
25 of 29 checks passed

strosek pushed a commit that referenced this pull request Apr 24, 2024

✨ [destination-DuckDB] Improve performance, use pyarrow batch inser…

842fbfe

…t as replacement of `executemany` (#36715)

FVidalCarneiro pushed a commit to AgiData/airbyte that referenced this pull request May 2, 2024

✨ [destination-DuckDB] Improve performance, use pyarrow batch inser…

deec837

…t as replacement of `executemany` (airbytehq#36715)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ [destination-DuckDB] use `pyarrow` batch insert to replace executemany #36715

✨ [destination-DuckDB] use `pyarrow` batch insert to replace executemany #36715

hrl20 commented Apr 1, 2024 •

edited

Loading

vercel bot commented Apr 1, 2024 •

edited

Loading

CLAassistant commented Apr 1, 2024 •

edited

Loading

aaronsteers left a comment

marcosmarxm commented Apr 8, 2024

aaronsteers left a comment

aaronsteers left a comment •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Apr 18, 2024

github-actions bot commented Apr 18, 2024

github-actions bot commented Apr 18, 2024 •

edited by aaronsteers

Loading

github-actions bot commented Apr 22, 2024 •

edited

Loading

github-actions bot commented Apr 22, 2024 •

edited

Loading

github-actions bot commented Apr 22, 2024 •

edited

Loading

aaronsteers commented Apr 22, 2024 •

edited

Loading

✨ [destination-DuckDB] use pyarrow batch insert to replace executemany #36715

✨ [destination-DuckDB] use pyarrow batch insert to replace executemany #36715

Conversation

hrl20 commented Apr 1, 2024 • edited Loading

What

How

User Impact

vercel bot commented Apr 1, 2024 • edited Loading

CLAassistant commented Apr 1, 2024 • edited Loading

aaronsteers left a comment

Choose a reason for hiding this comment

marcosmarxm commented Apr 8, 2024

aaronsteers left a comment

Choose a reason for hiding this comment

aaronsteers left a comment • edited Loading

Choose a reason for hiding this comment

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Apr 18, 2024

github-actions bot commented Apr 18, 2024

github-actions bot commented Apr 18, 2024 • edited by aaronsteers Loading

github-actions bot commented Apr 22, 2024 • edited Loading

github-actions bot commented Apr 22, 2024 • edited Loading

github-actions bot commented Apr 22, 2024 • edited Loading

aaronsteers commented Apr 22, 2024 • edited Loading

✨ [destination-DuckDB] use `pyarrow` batch insert to replace executemany #36715

✨ [destination-DuckDB] use `pyarrow` batch insert to replace executemany #36715

hrl20 commented Apr 1, 2024 •

edited

Loading

vercel bot commented Apr 1, 2024 •

edited

Loading

CLAassistant commented Apr 1, 2024 •

edited

Loading

aaronsteers left a comment •

edited

Loading

github-actions bot commented Apr 18, 2024 •

edited by aaronsteers

Loading

github-actions bot commented Apr 22, 2024 •

edited

Loading

github-actions bot commented Apr 22, 2024 •

edited

Loading

github-actions bot commented Apr 22, 2024 •

edited

Loading

aaronsteers commented Apr 22, 2024 •

edited

Loading