Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ESQL: Reorganize optimizer rules #112338

Merged

Conversation

alex-spies
Copy link
Contributor

@alex-spies alex-spies commented Aug 29, 2024

Closes #111317

Organize the optimizer rules consistently for all 4 optimizers (logical, physical, local logical, local physical) and move helper methods meant for optimizer rules out of the optimizers into the relevant rules or into helper classes. Make the optimizers now only contain code relevant to the actual optimizer definition, i.e. which batches of rules they use.

Also, consolidate the 2 nearly identical logical ParameterizedRules into one.

New package structure:

package org.elasticsearch.xpack.esql.optimizer
└── rules
    ├── logical
    │   ├── local
    └── physical
        └── local

New helper classes:

- org.elasticsearch.xpack.esql.optimizer.rules.logical.pushdown.PushDownUtils
- org.elasticsearch.xpack.esql.optimizer.rules.logical.TemporaryNameUtils
- org.elasticsearch.xpack.esql.optimizer.rules.physical.local.LucenePushDownUtils

@costin
Copy link
Member

costin commented Sep 4, 2024

Thanks for looking into this.
Quick feedback: at the logical level, the pushdown and local are on the same level which is for me confusing as local calls rules form the parent package as well; I suggest getting rid of the pushdown package and moving the place next to the rest of the other rules as there's nothing special about them.
lucenepushdown - quite the mouthful. At least at the moment it's the only time of pushdown possible in a local node and conceptually related to the other rules. I suggest to move these rules under local.

So the final structure looks something like:

package org.elasticsearch.xpack.esql.optimizer
└── rules
    ├── logical
    │   ├── local
    └── physical
        └── local

+1 on the separating the utils - even better make them protected.

@alex-spies alex-spies force-pushed the reorg-logical-plan-optimizer-rules branch from a778aa9 to 5148356 Compare September 4, 2024 10:14
@alex-spies alex-spies marked this pull request as ready for review September 4, 2024 10:14
@elasticsearchmachine elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Sep 4, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

@alex-spies alex-spies requested review from costin and astefan September 4, 2024 10:15
@alex-spies
Copy link
Contributor Author

alex-spies commented Sep 4, 2024

@costin, the structure is now the one you suggested + I made the new utils package private.

@alex-spies alex-spies changed the title ESQL: Reorganize optimizers ESQL: Reorganize optimizer rules Sep 4, 2024
Copy link
Member

@costin costin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM


import static org.elasticsearch.xpack.esql.common.Failure.fail;

public class DependencyConsistency<P extends QueryPlan<P>> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name should indicate the role of this class - checking the consistency of dependencies. Right now it sounds like a DTO class which contains the dependencies consistency.

How about something more mundane but clearer such as DependencyChecker, PlanStateChecker or ValidatePlanState?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class was just extracted from an already existent one, but it wouldn't hurt a rename tbh.

Thinking about the name, I realized we don't have a javadoc for it to explain its intention. How about adding a sentence there?

Leaving the javadoc aside and just looking at the code, I am throwing another name suggestion in the bucket, but up to @alex-spies to make the change or not altogether: PlanConsistencyChecker (the reasoning behind the name is that this is a checker class and does something, an action; it looks at the "inputs" and "outputs" of each node plan if there is an inconsistency there it spits out an error, thus the PlanConsistency part). My 2c.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went with Andrei's suggestion :)

Copy link
Contributor

@astefan astefan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks, Alex.

I've left some comments mostly spawning from personal preferences and experience. Feel free to adopt or ignore them.


import static org.elasticsearch.xpack.esql.common.Failure.fail;

public class DependencyConsistency<P extends QueryPlan<P>> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class was just extracted from an already existent one, but it wouldn't hurt a rename tbh.

Thinking about the name, I realized we don't have a javadoc for it to explain its intention. How about adding a sentence there?

Leaving the javadoc aside and just looking at the code, I am throwing another name suggestion in the bucket, but up to @alex-spies to make the change or not altogether: PlanConsistencyChecker (the reasoning behind the name is that this is a checker class and does something, an action; it looks at the "inputs" and "outputs" of each node plan if there is an inconsistency there it spits out an error, thus the PlanConsistency part). My 2c.

@alex-spies alex-spies merged commit 2f08d7d into elastic:main Sep 5, 2024
15 checks passed
@alex-spies alex-spies deleted the reorg-logical-plan-optimizer-rules branch September 5, 2024 08:55
@alex-spies
Copy link
Contributor Author

Thanks for the speedy reviews @astefan and @costin !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Analytics/ES|QL AKA ESQL >refactoring Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v8.16.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ESQL: remove cycle between LogicalPlanOptimizer and its rules
4 participants