chore: DataFusion 54.0.0

[comphead]

What is the problem the feature request solves?

The placeholder for DF 54.0.0

Describe the potential solution

No response

Additional context

No response

Internal dependencies

• use array_compact and array_repeat from Datafusion #3929

[comphead]

@mbutrovich cc

[mbutrovich]

I'm trying to maintain a branch for this release so Comet doesn't fall behind and isn't able to test x.0 RCs:

#3916

I update it every day or two, and file upstream fixes to DataFusion. So far it's found real issues that would have likely had us waiting for a x.1 or x.2 release.

Note that in the DF 54.0 branch I am now enabling SMJ with filter by default as the performance regressions there are now gone after apache/datafusion#20910 and apache/datafusion#21197.