Release DataFusion 37.0.0

[andygrove]

Is your feature request related to a problem or challenge?

It has been ~4 weeks since the last release was published. so I plan on starting the release process for the next version later this week.

Describe the solution you'd like

Waiting on:

• Regression: Wrong result when there are 2 count(distinct) #9586
• Regression: incorrect result for aliased arguments in select list after TreeNode rewrite #9678
• Update Arrow/Parquet to 51.0.0, tonic to 0.11 #9613
• Support backwards compatible / un prefixed config options (compression rather than parquet.compression) #9575
• Regression: All formatting options in COPY commands require format. prefix, but did not in DataFusion 36.0.0 #9716
• COPY TO fails on passing options for format through cli #9707
• Rename common_runtime directory to common-runtime for consistency with other crate directory names #9696
• Parallel parquet writer can panic when max_row_group_size < execution.batch_size #9736

Describe alternatives you've considered

No response

Additional context

No response

[alamb]

EPIC!

[alamb]

I think we should hold this release for two regression bugs:

1. Regression: Wrong result when there are 2 count(distinct) #9586
2. Regression: incorrect result for aliased arguments in select list after TreeNode rewrite #9678

I expect solutions to both by the end of day today

[leoyvens]

Any chance of including #9613? Seems desirable for DataFusion to tightly follow the latest arrow-rs dependencies.

[alamb]

Any chance of including #9613? Seems desirable for DataFusion to tightly follow the latest arrow-rs dependencies.

It makes sense to me. I'll go and get that PR ready for merge now

Update: merged

[alamb]

Something else we may want to wait for is backwards compatible write configuration options

• Support backwards compatible / un prefixed config options (compression rather than parquet.compression) #9575 (I think @tinfoil-knight is working on this)
• whatever fallout is needed to avoid format. prefix after Make COPY TO align with CREATE EXTERNAL TABLE #9604 (I am not sure there is a ticket tracking this yet)

cc @metesynnada and @ozankabak and @devinjdangelo

[alamb]

@wiedld has a PR up to fix #9678 ( #9685)

[alamb]

I have added the list of things I think we should wait on to the description of this ticket

[alamb]

I believe all outstanding items from my perspective are complete

[alamb]

@andygrove unless you or someone else does it first, I plan to make a release candidate tomorrow, Friday March 29. 2024

[alamb]

Vote thread is up: https://lists.apache.org/thread/lwdflob72q7t7mqqxnqcobjzqjbc218o

[alamb]

@wiedld has found another regression in 37.0.0: #9870 (see details in #9870 (comment))

Update: I reviewed the PR to fix it #9871 and it looks good to go to me

[alamb]

Update: I reviewed the PR to fix it #9871 and it looks good to go to me

Has been merged

[alamb]

RC2 voting thread: https://lists.apache.org/thread/mzjt0l4tzkm3y7hxsv8lk35x0xxq1q5b

[comphead]

Are we still planning to release 37.0.0?

[alamb]

I think we are now ready to go: the RC has been approved and 72 hours has elapsed.

https://lists.apache.org/thread/mzjt0l4tzkm3y7hxsv8lk35x0xxq1q5b

If @andygrove doesn't get a chance to publish to crates.io I can do it later today

[alamb]

The release is available here:
https://dist.apache.org/repos/dist/release/arrow/arrow-datafusion-37.0.0

The release has also been published to crates.io:
https://crates.io/crates/datafusion/37.0.0

Thank you all for your help!

Andrew

[alamb]

I made some small improvements based on my experience this time, but otherwise I think this is now done

1. Fix datafusion-cli publishing #9955
2. Minor: Update release README #9956

[alamb]

BTW we are making a 37.1.0 release -- see #9904

[alamb]

Note the failing CI test (security_audit) is explained here #10128 (comment).