Expose physical optimizer API

[andygrove]

Is your feature request related to a problem or challenge?

In DataFusion Comet, we are mapping Spark's physical plan to a DataFusion physical plan. I would like to be able to apply DataFusion's physical optimizer rules to this plan but there does not seem to be a public API for this and I could not find an example.

There is a public API in DefaultPhysicalPlanner to create an optimized physical plan from a logical plan:

/// Create a physical plan from a logical plan
async fn create_physical_plan(
    &self,
    logical_plan: &LogicalPlan,
    session_state: &SessionState,
) -> Result<Arc<dyn ExecutionPlan>> {

This calls the optimizer internally:

let plan = self
    .create_initial_plan(logical_plan, session_state)
    .await?;

self.optimize_internal(plan, session_state, |_, _| {})

The optimize_internal method seems to be what I need:

/// Optimize a physical plan by applying each physical optimizer,
/// calling observer(plan, optimizer after each one)
fn optimize_internal<F>(
    &self,
    plan: Arc<dyn ExecutionPlan>,
    session_state: &SessionState,
    mut observer: F,
) -> Result<Arc<dyn ExecutionPlan>>

Describe the solution you'd like

I would like to make optimize_internal public and change the name to something like optimize_physical_plan.

Describe alternatives you've considered

No response

Additional context

No response