[FLINK-30841][table-planner] Fix incorrect calc merge to avoid wrong plans

[lincoln-lil]

What is the purpose of the change

currently we have a FlinkCalcMergeRuleTest, take one test as example:

  @Test
  def testCalcMergeWithNonDeterministicExpr1(): Unit = {
    val sqlQuery = "SELECT a, a1 FROM (SELECT a, random_udf(a) AS a1 FROM MyTable) t WHERE a1 > 10"
    util.verifyRelPlan(sqlQuery)
  }

the current final optimized plan will be wrong:

Calc(select=[a, random_udf(b) AS a1], where=[(random_udf(b) > 10)])
+- LegacyTableSourceScan(table=[[default_catalog, default_database, MyTable, source: [TestTableSource(a, b, c)]]], fields=[a, b, c])

the merged calc contains two random_udf call, users may encounter the result satisfied by where predicate (>10) but the selected column <= 10, that's counter-intuitive for users

the expected plan is:

Calc(select=[a, a1], where=[(a1 > 10)])
+- Calc(select=[a, random_udf(b) AS a1])
   +- LegacyTableSourceScan(table=[[default_catalog, default_database, MyTable, source: [TestTableSource(a, b, c)]]], fields=[a, b, c])

Brief change log

• add a new rules related to this problem, FlinkFilterCalcMergeRule & FlinkFilterProjectTransposeRule

Verifying this change

Add new plan test CalcMergeTest

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (no)
• The public API, i.e., is any changed class annotated with @public(Evolving): (no)
• The serializers: (no )
• The runtime per-record code paths (performance sensitive): (no)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
• The S3 file system connector: (no)

Documentation

• Does this pull request introduce a new feature? (no)

[flinkbot]

CI report:

• e473a26 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[lincoln-lil]

I'll rebase the lastest master after reviewing, considering there'll be some adapting work to the new rule config changes(the new immutable annotation may cause some inconvenience in IDE) since calcite-1.29 upgrading has been done

[godfreyhe]

LGTM

[lincoln-lil]

rebased latest master include patches fixing the unstable tests