Improve DuckDB insert performance; support Postgres materialized views; MySQL connection metrics #311
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Add indexes mismatch detection for DuckDB table creation * Update src/duckdb.rs Co-authored-by: Luke Kim <80174+lukekim@users.noreply.github.com> * Update src/duckdb.rs Co-authored-by: Luke Kim <80174+lukekim@users.noreply.github.com> * Update src/duckdb.rs --------- Co-authored-by: Luke Kim <80174+lukekim@users.noreply.github.com>
* wip * refactor: New TableCreator interfaces for DuckDB * wip: Add some stubbed tests * wip: #[ignore] not #[skip] * test: Add creator.rs tests * test: Add overwrite and append tests * fix: Add schema validation back, add more unit tests * refactor: Rename TableCreator::list_internal_tables * chore: Remove commented out code * Switch from Appender to duckdb_arrow_scan (#288) * Switch from Appender to duckdb_arrow_scan * Fix duckdb-rs commit * Update src/duckdb.rs Co-authored-by: Luke Kim <80174+lukekim@users.noreply.github.com> --------- Co-authored-by: Luke Kim <80174+lukekim@users.noreply.github.com> * duckdb-rs commit from spiceai-1.1.3-backported * refactor: Append reuses existing base table, more tests, ViewCreator * fix: InsertOp::Replace needs Append * Add temp directory parameter (#289) * refactor: Make TableDefinition PartialEq all of the time * review: Rename TableCreator to TableManager, use option for internal table state * refactor: Update error messaging on indexes * Apply suggestions from code review Co-authored-by: Phillip LeBlanc <phillip@leblanc.tech> * fix: Address internal tables that are subsets or other table names * feat: Add TableDefinition function to determine if it exists * fix: Allow TableDefinition::name() to be public * clippy: must_use --------- Co-authored-by: Phillip LeBlanc <phillip@leblanc.tech> Co-authored-by: Luke Kim <80174+lukekim@users.noreply.github.com>
…r DuckDB (#290) * fix: Make initial table if none exist for DuckDB * refactor: Defer indexes to after first append load * test: Update tests for append * fix: Index detection for overwrite mode * clippy: Reference immediately dereferenced * test: Fix table creator tests
…y created tables. (#292)
…275) * Add connection_pool_size parameter for duckdb table provider * fix tests * remove user configured connection_pool param, add min_idle configuration to pool * fix * apply suggestions * fix tests * add DuckDBConnectionPoolBuilder method * remove idle size calculation, introduce non-breaking get_or_init instance methods * fix tests * Add mode to DuckDbConnectionPoolBuilder, remove lifetime for DuckDBConnectionPoolBuilder * move build pool functionality into the DuckDB ConnectionPool Builder * Use builder when get_or_init instance * only pass pool to the get_or_init_instance_with_builder * remove unnecessary error
…rialized views (#300) * Update the PostgreSQL query to support inferring the schema from materialized views * update postgres schema tests, to use complex table for views * Update examples/postgres.rs * Update examples/postgres.rs
…eters (#302) * Upgrade mysql_async, expose metrics and remove unnecessary type parameters * Fix lint issue
* Support the `pool_min` and `pool_max` parameters * Support the `pool_min` and `pool_max` MySQL connection parameters * Exercise the pool_min/pool_max options
ewgenius
approved these changes
Apr 10, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Merges in fixes made to the
spiceaibranch that cover the following three main areas:Appendertoduckdb_arrow_scanwhich leverages DuckDB zero-copy of Arrow. Unfortunately, this required forking theduckdbcrate to exposeduckdb_arrow_scansince the upstream PR will not be merged anytime soon (see Implement support forduckdb_arrow_scanfor ingesting data via Arrow duckdb/duckdb-rs#488)