Make OptimizerConfig a trait (#4631) (#4638) #4645

tustvold · 2022-12-15T11:43:35Z

Which issue does this PR close?

Closes #4631
Closes #4638

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

tustvold · 2022-12-15T11:44:34Z

datafusion/core/src/execution/context.rs

@@ -1557,14 +1557,6 @@ impl SessionState {
                .register_catalog(config.default_catalog.clone(), default_catalog);
        }

-        let optimizer_config = OptimizerConfig::new().filter_null_keys(


Moving this into the OptimizerConfig passed at optimize time is the fix for #4638

tustvold · 2022-12-15T11:45:30Z

datafusion/optimizer/src/optimizer.rs

        ];
-        if config.filter_null_keys {


This is the other half of the fix for #4638 - determine the rules dynamically

tustvold · 2022-12-15T11:46:17Z

datafusion/optimizer/src/optimizer.rs

+    fn query_execution_start_time(&self) -> DateTime<Utc>;
+
+    /// Returns false if the given rule should be skipped
+    fn rule_enabled(&self, name: &str) -> bool;


I felt this formulation was more future proof than adding a specific function for enabling/disabling FilterNullJoins

tustvold · 2022-12-15T11:46:51Z

datafusion/optimizer/src/optimizer.rs

    /// Create optimizer config
    pub fn new() -> Self {
        Self {
            query_execution_start_time: Utc::now(),
-            next_id: 0, // useful for generating things like unique subquery aliases
+            next_id: AtomicUsize::new(1),


We use 1, as previously the implementation returned the value post-increment which the atomics don't provide

tustvold · 2022-12-15T11:47:35Z

datafusion/optimizer/src/optimizer.rs

-    // TODO this should not be on the config,
-    // it should be its own 'OptimizerState' or something)
-    next_id: usize,
+    next_id: AtomicUsize,


I think this limited interior mutability is better than the potential confusion around rules taking a mutable OptimizerConfig

I agree -- cc @waynexia

It is also good that OptimizerContext doesn't implement Clone which will discourage making copies that could get out of date

Looks good 👍

tustvold · 2022-12-15T11:48:44Z

datafusion/core/src/execution/context.rs

-            .with_query_execution_start_time(
-                self.execution_props.query_execution_start_time,
-            );
+        // TODO: Implement OptimizerContext directly on DataFrame (#4631) (#4626)


This is the ultimate motivation for this change, to allow a single config container, that then just implements the traits needed by the various sub-systems

I think having the sub crates depend on a traits that are implemented in the core trait sounds like a good idea to me

jackwener · 2022-12-15T13:38:59Z

I have some PR about optimizer😂, look like it will make some conflict.

I will wait for this PR merge, and then do them.

alamb

Looks like a (really) nice improvement to me -- thank you @tustvold 🏆

I think we should gather some more input and leave this open a while before merging it even though I think the actual required API change downstream are relatively low

cc @andygrove @jackwener @yahoNanJing @Dandandan @thinkharderdev @xudong963

alamb · 2022-12-15T18:38:30Z

datafusion/core/src/execution/context.rs

-            .with_query_execution_start_time(
-                self.execution_props.query_execution_start_time,
-            );
+        // TODO: Implement OptimizerContext directly on DataFrame (#4631) (#4626)


I think having the sub crates depend on a traits that are implemented in the core trait sounds like a good idea to me

alamb · 2022-12-15T18:41:31Z

datafusion/core/src/execution/context.rs

@@ -1557,14 +1557,6 @@ impl SessionState {
                .register_catalog(config.default_catalog.clone(), default_catalog);
        }

-        let optimizer_config = OptimizerConfig::new().filter_null_keys(


alamb · 2022-12-15T18:43:55Z

datafusion/optimizer/src/optimizer.rs

-    // TODO this should not be on the config,
-    // it should be its own 'OptimizerState' or something)
-    next_id: usize,
+    next_id: AtomicUsize,


I agree -- cc @waynexia

alamb · 2022-12-15T18:44:59Z

datafusion/optimizer/src/optimizer.rs

-    // TODO this should not be on the config,
-    // it should be its own 'OptimizerState' or something)
-    next_id: usize,
+    next_id: AtomicUsize,


It is also good that OptimizerContext doesn't implement Clone which will discourage making copies that could get out of date

datafusion/optimizer/src/optimizer.rs

xudong963

The change makes sense to me, thanks @tustvold

thinkharderdev

Makes sense to me

…trait

tustvold · 2022-12-16T15:24:39Z

I plan to merge this in the next few hours unless anybody objects / needs more time to review

ursabot · 2022-12-16T18:32:09Z

Benchmark runs are scheduled for baseline = 8944581 and contender = ca8985e. ca8985e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

alamb · 2022-12-16T19:35:22Z

🎉

really nice work @tustvold

tustvold added the api change Changes the API exposed to users of the crate label Dec 15, 2022

github-actions bot added core Core DataFusion crate optimizer Optimizer rules labels Dec 15, 2022

tustvold commented Dec 15, 2022

View reviewed changes

tustvold requested a review from alamb December 15, 2022 11:48

Make OptimizerConfig a trait (apache#4631) (apache#4638)

babfb99

tustvold force-pushed the optimizer-config-trait branch from 8ce5714 to babfb99 Compare December 15, 2022 12:48

Format

367c21e

alamb approved these changes Dec 15, 2022

View reviewed changes

xudong963 approved these changes Dec 16, 2022

View reviewed changes

thinkharderdev approved these changes Dec 16, 2022

View reviewed changes

tustvold added 2 commits December 16, 2022 15:22

Review feedback

bd9b3c9

Merge remote-tracking branch 'upstream/master' into optimizer-config-…

dbbcff5

…trait

tustvold merged commit ca8985e into apache:master Dec 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make OptimizerConfig a trait (#4631) (#4638) #4645

Make OptimizerConfig a trait (#4631) (#4638) #4645

tustvold commented Dec 15, 2022 •

edited

Loading

tustvold Dec 15, 2022

alamb Dec 15, 2022

tustvold Dec 15, 2022

tustvold Dec 15, 2022

tustvold Dec 15, 2022

tustvold Dec 15, 2022

alamb Dec 15, 2022

alamb Dec 15, 2022

waynexia Dec 16, 2022

tustvold Dec 15, 2022

alamb Dec 15, 2022

jackwener commented Dec 15, 2022 •

edited

Loading

alamb left a comment

alamb Dec 15, 2022

alamb Dec 15, 2022

alamb Dec 15, 2022

alamb Dec 15, 2022

xudong963 left a comment

thinkharderdev left a comment

tustvold commented Dec 16, 2022

ursabot commented Dec 16, 2022

alamb commented Dec 16, 2022

Make OptimizerConfig a trait (#4631) (#4638) #4645

Make OptimizerConfig a trait (#4631) (#4638) #4645

Conversation

tustvold commented Dec 15, 2022 • edited Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jackwener commented Dec 15, 2022 • edited Loading

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xudong963 left a comment

Choose a reason for hiding this comment

thinkharderdev left a comment

Choose a reason for hiding this comment

tustvold commented Dec 16, 2022

ursabot commented Dec 16, 2022

alamb commented Dec 16, 2022

tustvold commented Dec 15, 2022 •

edited

Loading

jackwener commented Dec 15, 2022 •

edited

Loading