Skip to content

Commit

Permalink
update changelog
Browse files Browse the repository at this point in the history
  • Loading branch information
@andygrove
andygrove committed Mar 28, 2024
1 parent 1b611ea commit 42af516
Showing 1 changed file with 44 additions and 83 deletions.
127 changes: 44 additions & 83 deletions dev/changelog/37.0.0.md
View file
Original file line number Diff line number Diff line change
@@ -1,29 +1,5 @@
<!---
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

## [37.0.0](https://github.com/apache/arrow-datafusion/tree/37.0.0) (2024-03-19)

[Full Changelog](https://github.com/apache/arrow-datafusion/compare/36.0.0...37.0.0)

**Breaking changes:**

- refactor: Fix typo: Rename`PruningPredicate::allways_true()` to `PruningPredicate::always_true()` [#9283](https://github.com/apache/arrow-datafusion/pull/9283) (SteveLauC)
- refactor: Change `SchemaProvider::table` to return `Result<Option<..>` rather than `Option<..>` [#9307](https://github.com/apache/arrow-datafusion/pull/9307) (crepererum)
- feat: issue_9285: port builtin reg function into datafusion-function-\* (1/3 regexpmatch) [#9329](https://github.com/apache/arrow-datafusion/pull/9329) (Lordworms)
- Cache common plan properties to eliminate recursive calls in physical plan [#9346](https://github.com/apache/arrow-datafusion/pull/9346) (mustafasrepo)
Expand All @@ -34,14 +10,10 @@

**Performance related:**

- Implement specialized group values for single Uft8/LargeUtf8/Binary/LargeBinary column [#8827](https://github.com/apache/arrow-datafusion/pull/8827) (alamb)
- perf: improve to_field performance [#9722](https://github.com/apache/arrow-datafusion/pull/9722) (haohuaijin)

**Implemented enhancements:**

- feat: Implement `FunctionRegistry::register_udwf` and `FunctionRegistry::register_udaf` for SessionContext, MemoryFunctionRegistry and TaskContext [#9226](https://github.com/apache/arrow-datafusion/pull/9226) (Lordworms)
- feat: support `FixedSizeList` Type Coercion [#9108](https://github.com/apache/arrow-datafusion/pull/9108) (Weijun-H)
- feat: expand `unnest` to accept arbitrary single array expression [#9342](https://github.com/apache/arrow-datafusion/pull/9342) (jonahgao)
- feat: support for defining ARRAY columns in `CREATE TABLE` [#9381](https://github.com/apache/arrow-datafusion/pull/9381) (jonahgao)
- feat: support `unnest` in FROM clause [#9355](https://github.com/apache/arrow-datafusion/pull/9355) (jonahgao)
- feat: support nvl2 function [#9364](https://github.com/apache/arrow-datafusion/pull/9364) (guojidan)
Expand All @@ -56,16 +28,13 @@
- feat: implement more expr_to_sql functionality [#9578](https://github.com/apache/arrow-datafusion/pull/9578) (devinjdangelo)
- feat: implement aggregation and subquery plans to SQL [#9606](https://github.com/apache/arrow-datafusion/pull/9606) (devinjdangelo)
- feat: track memory usage for recursive CTE, enable recursive CTEs by default [#9619](https://github.com/apache/arrow-datafusion/pull/9619) (jonahgao)
- feat: Between expr to sql string [#9803](https://github.com/apache/arrow-datafusion/pull/9803) (sebastian2296)
- feat: Expose `array_empty` and `list_empty` functions as alias of `empty` function [#9807](https://github.com/apache/arrow-datafusion/pull/9807) (erenavsarogullari)
- feat: Not expr to string [#9802](https://github.com/apache/arrow-datafusion/pull/9802) (sebastian2296)
- feat: pass SessionState not SessionConfig to FunctionFactory::create [#9837](https://github.com/apache/arrow-datafusion/pull/9837) (tshauck)

**Fixed bugs:**

- fix: issue #9130 substitute redundant columns when doing cross join [#9154](https://github.com/apache/arrow-datafusion/pull/9154) (Lordworms)
- fix: fix `GLIBC_2.35' not found error in ci [#9243](https://github.com/apache/arrow-datafusion/pull/9243) (ZhengLin-Li)
- fix: add `ArrayAndElementAndOptionalIndex` for proper casting in `array_position` [#9233](https://github.com/apache/arrow-datafusion/pull/9233) (tshauck)
- fix: issue #8838 discard extra sort when sorted element is wrapped [#9127](https://github.com/apache/arrow-datafusion/pull/9127) (Lordworms)
- fix: issue #9213 substitute ArrayAgg to NthValue to optimize query plan [#9295](https://github.com/apache/arrow-datafusion/pull/9295) (Lordworms)
- fix: throw error when incursion happen in dataframe api [#9330](https://github.com/apache/arrow-datafusion/pull/9330) (Lordworms)
- fix: flight examples [#9335](https://github.com/apache/arrow-datafusion/pull/9335) (codonnell)
- fix: use `JoinSet` to make spawned tasks cancel-safe [#9318](https://github.com/apache/arrow-datafusion/pull/9318) (DDtKey)
- fix: nvl function's return type [#9357](https://github.com/apache/arrow-datafusion/pull/9357) (guojidan)
- fix: panic in isnan() when no args are given [#9377](https://github.com/apache/arrow-datafusion/pull/9377) (SteveLauC)
Expand All @@ -80,14 +49,11 @@
- fix: incorrect null handling in `range` and `generate_series` [#9574](https://github.com/apache/arrow-datafusion/pull/9574) (jonahgao)
- fix: recursive cte hangs on joins [#9687](https://github.com/apache/arrow-datafusion/pull/9687) (jonahgao)
- fix: parallel parquet can underflow when max_record_batch_rows < execution.batch_size [#9737](https://github.com/apache/arrow-datafusion/pull/9737) (devinjdangelo)
- fix: change placeholder errors from Internal to Plan [#9745](https://github.com/apache/arrow-datafusion/pull/9745) (erratic-pattern)
- fix: ensure mutual compatibility of the two input schemas from recursive CTEs [#9795](https://github.com/apache/arrow-datafusion/pull/9795) (jonahgao)

**Documentation updates:**

- Prepare 36.0.0-rc1 [#9251](https://github.com/apache/arrow-datafusion/pull/9251) (andygrove)
- Update Docs with Copy partition_by support [#9275](https://github.com/apache/arrow-datafusion/pull/9275) (devinjdangelo)
- docs: update contributor guide (format toml/inte test) [#9301](https://github.com/apache/arrow-datafusion/pull/9301) (SteveLauC)
- docs: document range() alias generate_series() [#9321](https://github.com/apache/arrow-datafusion/pull/9321) (SteveLauC)
- docs: update parquet_sql_multiple_files.rs with a relative path ex [#9310](https://github.com/apache/arrow-datafusion/pull/9310) (tshauck)
- docs: put flatten in top fn list [#9376](https://github.com/apache/arrow-datafusion/pull/9376) (SteveLauC)
- Update documentation so list_to_string alias to point to array_to_string [#9374](https://github.com/apache/arrow-datafusion/pull/9374) (monkwire)
- Uplift keys/dependencies to use more workspace inheritance [#9293](https://github.com/apache/arrow-datafusion/pull/9293) (Jefffrey)
Expand All @@ -106,49 +72,6 @@

**Merged pull requests:**

- Prepare 36.0.0-rc1 [#9251](https://github.com/apache/arrow-datafusion/pull/9251) (andygrove)
- fix: issue #9130 substitute redundant columns when doing cross join [#9154](https://github.com/apache/arrow-datafusion/pull/9154) (Lordworms)
- minor: Add more doc to clarify permutation satisfying input order in `InputOrderMode` [#9255](https://github.com/apache/arrow-datafusion/pull/9255) (viirya)
- fix: fix `GLIBC_2.35' not found error in ci [#9243](https://github.com/apache/arrow-datafusion/pull/9243) (ZhengLin-Li)
- fix: add `ArrayAndElementAndOptionalIndex` for proper casting in `array_position` [#9233](https://github.com/apache/arrow-datafusion/pull/9233) (tshauck)
- Support Copy To Partitioned Files [#9240](https://github.com/apache/arrow-datafusion/pull/9240) (devinjdangelo)
- Minior: Add negative tests about log function [#9245](https://github.com/apache/arrow-datafusion/pull/9245) (caicancai)
- Minor: fix warning of 'Unnecessary braces in use statement' [#9271](https://github.com/apache/arrow-datafusion/pull/9271) (jonahgao)
- Remove circular dependency [#9279](https://github.com/apache/arrow-datafusion/pull/9279) (andygrove)
- chore: use taplo to format Cargo.toml manifests [#9263](https://github.com/apache/arrow-datafusion/pull/9263) (waynexia)
- feat: Implement `FunctionRegistry::register_udwf` and `FunctionRegistry::register_udaf` for SessionContext, MemoryFunctionRegistry and TaskContext [#9226](https://github.com/apache/arrow-datafusion/pull/9226) (Lordworms)
- Implement specialized group values for single Uft8/LargeUtf8/Binary/LargeBinary column [#8827](https://github.com/apache/arrow-datafusion/pull/8827) (alamb)
- Fix untracked files after rustdoc run [#9264](https://github.com/apache/arrow-datafusion/pull/9264) (Omega359)
- Support List for Array aggregate order and distinct [#9234](https://github.com/apache/arrow-datafusion/pull/9234) (jayzhan211)
- fix: issue #8838 discard extra sort when sorted element is wrapped [#9127](https://github.com/apache/arrow-datafusion/pull/9127) (Lordworms)
- refactor: Fix typo: Rename`PruningPredicate::allways_true()` to `PruningPredicate::always_true()` [#9283](https://github.com/apache/arrow-datafusion/pull/9283) (SteveLauC)
- Improve datafuson-proto and datafusion-substrait docs [#9260](https://github.com/apache/arrow-datafusion/pull/9260) (alamb)
- Minor: Add ticket reference to test [#9270](https://github.com/apache/arrow-datafusion/pull/9270) (alamb)
- Update Docs with Copy partition_by support [#9275](https://github.com/apache/arrow-datafusion/pull/9275) (devinjdangelo)
- Continuing Work of [PR9127]: improved using existing orderings [#9273](https://github.com/apache/arrow-datafusion/pull/9273) (mustafasrepo)
- Fix partition columns when they they are not at the end of the schema (fix Removing Partition Columns from RecordBatch) [#9276](https://github.com/apache/arrow-datafusion/pull/9276) (devinjdangelo)
- Minor: rename datafusion/proto from `README-dev.md` to `CONTRIBUTING.md` [#9300](https://github.com/apache/arrow-datafusion/pull/9300) (alamb)
- Fix array_ndims error handling [#9272](https://github.com/apache/arrow-datafusion/pull/9272) (Weijun-H)
- Fix toml format script [#9309](https://github.com/apache/arrow-datafusion/pull/9309) (Omega359)
- docs: update contributor guide (format toml/inte test) [#9301](https://github.com/apache/arrow-datafusion/pull/9301) (SteveLauC)
- [MINOR]: Limit stream replace with slice [#9303](https://github.com/apache/arrow-datafusion/pull/9303) (mustafasrepo)
- Support IGNORE NULLS for LAG window function [#9221](https://github.com/apache/arrow-datafusion/pull/9221) (comphead)
- fix: issue #9213 substitute ArrayAgg to NthValue to optimize query plan [#9295](https://github.com/apache/arrow-datafusion/pull/9295) (Lordworms)
- Minor: Adding missing fields to debug for `ScalarFunctionExpr` [#9325](https://github.com/apache/arrow-datafusion/pull/9325) (comphead)
- docs: document range() alias generate_series() [#9321](https://github.com/apache/arrow-datafusion/pull/9321) (SteveLauC)
- chore: statically link xz2 [#9311](https://github.com/apache/arrow-datafusion/pull/9311) (universalmind303)
- fix: throw error when incursion happen in dataframe api [#9330](https://github.com/apache/arrow-datafusion/pull/9330) (Lordworms)
- Support CopyTo::partition_by in datafusion proto [#9306](https://github.com/apache/arrow-datafusion/pull/9306) (PhVHoang)
- Add test to prevent circular dependencies from being added [#9292](https://github.com/apache/arrow-datafusion/pull/9292) (andygrove)
- Substrait: Support ScalarUDF [#9337](https://github.com/apache/arrow-datafusion/pull/9337) (jayzhan211)
- Make agg_func_substitute test deterministic [#9340](https://github.com/apache/arrow-datafusion/pull/9340) (andygrove)
- feat: support `FixedSizeList` Type Coercion [#9108](https://github.com/apache/arrow-datafusion/pull/9108) (Weijun-H)
- feat: expand `unnest` to accept arbitrary single array expression [#9342](https://github.com/apache/arrow-datafusion/pull/9342) (jonahgao)
- fix: flight examples [#9335](https://github.com/apache/arrow-datafusion/pull/9335) (codonnell)
- docs: update parquet_sql_multiple_files.rs with a relative path ex [#9310](https://github.com/apache/arrow-datafusion/pull/9310) (tshauck)
- tests: add tests for writing hive-partitioned parquet [#9316](https://github.com/apache/arrow-datafusion/pull/9316) (tshauck)
- feature: support nvl(ifnull) function [#9284](https://github.com/apache/arrow-datafusion/pull/9284) (guojidan)
- Move abs to datafusion_functions [#9313](https://github.com/apache/arrow-datafusion/pull/9313) (yyy1000)
- refactor: Change `SchemaProvider::table` to return `Result<Option<..>` rather than `Option<..>` [#9307](https://github.com/apache/arrow-datafusion/pull/9307) (crepererum)
- fix write_partitioned_parquet_results test case bug [#9360](https://github.com/apache/arrow-datafusion/pull/9360) (guojidan)
- fix: use `JoinSet` to make spawned tasks cancel-safe [#9318](https://github.com/apache/arrow-datafusion/pull/9318) (DDtKey)
Expand Down Expand Up @@ -361,3 +284,41 @@
- support format in options of COPY command [#9744](https://github.com/apache/arrow-datafusion/pull/9744) (tinfoil-knight)
- Move lower, octet_length to datafusion-functions [#9747](https://github.com/apache/arrow-datafusion/pull/9747) (Omega359)
- Fixed missing trim() in rust api [#9749](https://github.com/apache/arrow-datafusion/pull/9749) (Omega359)
- refactor: Extract `array_length`, `array_reverse` and `array_sort` functions from `functions-array` subcrate' s `kernels` and `udf` containers [#9751](https://github.com/apache/arrow-datafusion/pull/9751) (erenavsarogullari)
- refactor: Extract `array_empty` and `array_repeat` functions from `functions-array` subcrate' s `kernels` and `udf` containers [#9762](https://github.com/apache/arrow-datafusion/pull/9762) (erenavsarogullari)
- Minor: remove an outdated TODO in `TypeCoercion` [#9752](https://github.com/apache/arrow-datafusion/pull/9752) (jonahgao)
- refactor: Extract `array_resize` and `cardinality` functions from `functions-array` subcrate' s `kernels` and `udf` containers [#9766](https://github.com/apache/arrow-datafusion/pull/9766) (erenavsarogullari)
- fix: change placeholder errors from Internal to Plan [#9745](https://github.com/apache/arrow-datafusion/pull/9745) (erratic-pattern)
- Move levenshtein, uuid, overlay to datafusion-functions [#9760](https://github.com/apache/arrow-datafusion/pull/9760) (Omega359)
- improve null handling for to_char [#9689](https://github.com/apache/arrow-datafusion/pull/9689) (tinfoil-knight)
- Add Expr->String for ScalarFunction and InList [#9759](https://github.com/apache/arrow-datafusion/pull/9759) (yyy1000)
- Move repeat, replace, split_part to datafusion_functions [#9784](https://github.com/apache/arrow-datafusion/pull/9784) (Omega359)
- refactor: Extract `array_dims`, `array_ndims` and `flatten` functions from `functions-array` subcrate' s `kernels` and `udf` containers [#9786](https://github.com/apache/arrow-datafusion/pull/9786) (erenavsarogullari)
- Minor: Improve documentation about `ColumnarValues::values_to_array` [#9774](https://github.com/apache/arrow-datafusion/pull/9774) (alamb)
- Fix panic in `struct` function with mixed scalar/array arguments [#9775](https://github.com/apache/arrow-datafusion/pull/9775) (alamb)
- refactor: Apply minor refactorings to `functions-array` crate [#9788](https://github.com/apache/arrow-datafusion/pull/9788) (erenavsarogullari)
- Move bit_length and chr functions to datafusion_functions [#9782](https://github.com/apache/arrow-datafusion/pull/9782) (PsiACE)
- Support tencent cloud COS storage in `datafusion-cli` [#9734](https://github.com/apache/arrow-datafusion/pull/9734) (harveyyue)
- Make it easier to register configuration extension ... [#9781](https://github.com/apache/arrow-datafusion/pull/9781) (milenkovicm)
- Expr to Sql : Case [#9798](https://github.com/apache/arrow-datafusion/pull/9798) (yyy1000)
- feat: Between expr to sql string [#9803](https://github.com/apache/arrow-datafusion/pull/9803) (sebastian2296)
- feat: Expose `array_empty` and `list_empty` functions as alias of `empty` function [#9807](https://github.com/apache/arrow-datafusion/pull/9807) (erenavsarogullari)
- Support Expr `Like` to sql [#9805](https://github.com/apache/arrow-datafusion/pull/9805) (Weijun-H)
- feat: Not expr to string [#9802](https://github.com/apache/arrow-datafusion/pull/9802) (sebastian2296)
- [Minor]: Move some repetitive codes to functions(proto) [#9811](https://github.com/apache/arrow-datafusion/pull/9811) (mustafasrepo)
- Implement IGNORE NULLS for LAST_VALUE [#9801](https://github.com/apache/arrow-datafusion/pull/9801) (huaxingao)
- [MINOR]: Move some repetitive codes to functions [#9810](https://github.com/apache/arrow-datafusion/pull/9810) (mustafasrepo)
- fix: ensure mutual compatibility of the two input schemas from recursive CTEs [#9795](https://github.com/apache/arrow-datafusion/pull/9795) (jonahgao)
- Add support for constant expression evaluation in limit [#9790](https://github.com/apache/arrow-datafusion/pull/9790) (mustafasrepo)
- Projection Pushdown through user defined LogicalPlan nodes. [#9690](https://github.com/apache/arrow-datafusion/pull/9690) (mustafasrepo)
- chore(deps): update substrait requirement from 0.27.0 to 0.28.0 [#9809](https://github.com/apache/arrow-datafusion/pull/9809) (dependabot[bot])
- Run TPC-H SF10 during PR benchmarks [#9822](https://github.com/apache/arrow-datafusion/pull/9822) (gruuya)
- Expose `parser` on DFParser to enable user controlled parsing [#9729](https://github.com/apache/arrow-datafusion/pull/9729) (tshauck)
- Disable parallel reading for gziped ndjson file [#9799](https://github.com/apache/arrow-datafusion/pull/9799) (Lordworms)
- Optimize to_timestamp (with format) (#9090) [#9833](https://github.com/apache/arrow-datafusion/pull/9833) (vojtechtoman)
- Create unicode module in datafusion/functions/src/unicode and unicode_expressions feature flag, move char_length function [#9825](https://github.com/apache/arrow-datafusion/pull/9825) (Omega359)
- [Minor] Update TCPDS tests, remove some #[ignore]d tests [#9829](https://github.com/apache/arrow-datafusion/pull/9829) (Dandandan)
- doc: Adding baseline benchmark example [#9827](https://github.com/apache/arrow-datafusion/pull/9827) (comphead)
- Add name method to execution plan [#9793](https://github.com/apache/arrow-datafusion/pull/9793) (matthewmturner)
- chore(deps-dev): bump express from 4.18.2 to 4.19.2 in /datafusion/wasmtest/datafusion-wasm-app [#9826](https://github.com/apache/arrow-datafusion/pull/9826) (dependabot[bot])
- feat: pass SessionState not SessionConfig to FunctionFactory::create [#9837](https://github.com/apache/arrow-datafusion/pull/9837) (tshauck)

0 comments on commit 42af516

Please sign in to comment.