Allow connectors to participate in query optimization

[martint]

This is an umbrella issue to track various projects related to allowing connectors to participate in query optimization. The long-term vision is for plugins to provide custom rules that can transform subplans into plugin-specific operations. This requires a set of steps:

• decouple the AST from the current IR (PlanNode tree): Split expression ST from IR #13184
• get rid of visitor-based optimizers
• revamp the IR and optimizer to support a fully exploratory optimizer
• allow connectors to provide optimizer rules

In the short term, we can introduce special-purpose mechanisms to the SPI and engine enable the following behaviors:

• push down complex filters
• push down projections (e.g, row/array/map dereference, pre-calculated virtual columns)
• push down aggregations
• push down joins
• expose additional filters (e.g. for row-level authorization)
• expose more complex data organizations (e.g., custom partitioning schemes)

Document describing the high-level approach: https://github.com/prestosql/presto/wiki/Pushdown-of-complex-operations

Plan:

• Split expression ST from IR #13184
• Simplify and hide notion of TableLayout: Hide table layouts from engine #363
• Deprecate table layouts: Deprecate table layouts #420
  • Support Constraint-based pushdown to transition from Table Layout: Add applyFilter equivalent to Constraint-based table layout selection #541
• Define simplified expression language (some work in progress: [WIP] Complex pushdown  #402)
• Provide TableHandle to PageSource/RecordCursor providers: single TupleDomain constraint for multiple splits #442
• Add pushXXXIntoConnector + Rule pairs
  • Limit: Add support for limit pushdown into connectors #421
  • Filter: [WIP] Complex pushdown  #402, ConnectorExpression pushdown #7994
    • Use canonical function name in expression pushdown (e.g. pass substring instead of substr)
  • Projection: Projection pushdown #676
  • Aggregation (Aggregation pushdown #6613)
  • Join (Join pushdown #6620)
  • Sample: Add support for sample pushdown #753
  • Top N
    • engine & SPI (SPI and optimizer rule for connectors that can support complete topN … #4249)
    • JDBC connectors Implement TopN pushdown for JDBC connectors #4769
    • Push PARTIAL TopN into TableScan Push PARTIAL TopN into TableScan #7028
    • Push TopN through Project into TableScan rule Push TopN through Project into TableScan rule #7029
    • Elasticsearch Elasticsearch connector TopN pushdown #4803
• Integrate into connectors
  • Hive
    • Parquet + ORC - Migrate [WIP] Pushdown dereference expression to Parquet reader #187 to new framework
    • Optimize Top N queries over partition keys Optimize 'SELECT ... ORDER BY partition LIMIT ...' query #3050
  • JDBC connectors
    • aggregation (Aggregation pushdown #6613)
    • join (Join pushdown #6620)
      • ...
• Remove TableLayout-related APIs: Remove table layouts from all connectors #781
• SchemaTableName should not be required for synthetic ConnectorTableHandle returned from ConnectorMetadata.apply methods SchemaTableName should not be required for synthetic ConnectorTableHandle returned from ConnectorMetadata.apply methods #6694
• Support connectors advanced pushdown in EXPLAIN IO Support connectors advanced pushdown in EXPLAIN IO #6695

[sopel39]

revamp the IR and optimizer to support a fully exploratory optimizer

Related to exploratory optimizer is a concept of traits. I'm appending some old notes:
Traits roadmap:

• Add Lookup#resolveGroup method and make Lookup#resolve depricated prestodb/presto#8698 (comment)
• Migrate AddExchanges to iterative optimizer prestodb/presto#7169
  Design "proposal" how to use traits for rule programming: https://docs.google.com/presentation/d/10wCmZEp5NnRSb_4oPsdiKkmYy4Hy5BgddEA8brPuVis/edit?usp=sharing

[sopel39]

In addition, some worthy refactoring proposals that came from discussions around prestodb/presto#11591:
prestodb/presto#11591 (comment)
prestodb/presto#11689
FYI: @findepi

[pgagnon]

related to:
prestodb/presto#11777
prestodb/presto#4839

[martint]

Also related: #187

[martint]

I've started collecting my thoughts on how I think we should approach this in this documents: https://github.com/prestosql/presto/wiki/Pushdown-of-complex-operations. It's still incomplete and lacking many details, but the overall direction shouldn't have to change much.