Support multiple planner output dialects

[ryannedolan]

The planner is currently hard-coded to output MySQL dialect, which Flink natively understands. In order to target additional runtimes, we need a way to generate SQL in other dialects. N.B. the planner is largely dialect-agnostic already, until you ask for the plan as a String.

[HoganEdwardChu]

Hi, I didn't mean to disturb you, I was watching with interest. The line that you mean is here?

Hoptimator/hoptimator-planner/src/main/java/com/linkedin/hoptimator/planner/PipelineRel.java

Line 39 in f748776

SqlDialect OUTPUT_DIALECT = MysqlSqlDialect.DEFAULT; // closely resembles Flink SQL

[ryannedolan]

The line that you mean is here?

Yep. It looks like the hard-coded dialect is only used in a couple methods that return SQL strings. It might make sense to change those to return the underlying ScriptImplementer, which is dialect agnostic. Then callers could do rel.query(...).sql(dialect) and get whatever dialect they want.

[HoganEdwardChu]

Oh you mean https://github.com/HoganEdwardChu/Hoptimator/pull/1/files this way? I haven't seen all the code, so I don't know the specific details, so it might seem lacking(sorry for that) But than i think we have to get input dialect.

[ryannedolan]

this way?

Exactly, tho I'm not sure where to go from there. One way would be to add a dialect parameter to pipeline(), s.t. the operator can generate pipelines in whatever dialect it wants. For now, the operator can just use the MySQL dialect.

Looking ahead, we'd like to be able to implement alternative runtimes, e.g. using Spark SQL instead of Flink SQL. Ideally, those changes would be in the operator, not the planner. So this would be a step in that direction.

[HoganEdwardChu]

Thanks for the detailed and contextual explanation. I'll think about what you said too. (if there's a good direction, I'll try it and share it again too!) Also would try implementing alternative runtimes Spark SQL instead of Flink SQL.(operator part)

[ryannedolan]

I haven't given much thought to how we'd support multiple runtimes in the operator. There is a flink adapter, which is really the only place in the control plane that we assume the SQL will run on flink: https://github.com/linkedin/Hoptimator/blob/main/hoptimator-flink-adapter/src/main/java/com/linkedin/hoptimator/operator/flink/FlinkSqlJobReconciler.java#L57

As you can see, the flink adapter just passes the SQL from SqlJob to a FlinkDeployment (which is handled by the external flink-kubernetes-operator). Theoretically, we could do something similar for Spark or any other SQL runtime. I'm not sure how valuable that would be, but it seems possible.

[HoganEdwardChu]

I guess I'll have to think about it a bit. If you have any good opinions on whether it is better to put a dialect in the pipeline, please proceed. I will continue to try. I think making another adapter should be postponed to a later date. I think the most important thing is to handle the flink job well.

[HoganEdwardChu]

I think adding sqlDialect to pipeline is best ... maybe at this point
https://github.com/HoganEdwardChu/Hoptimator/blob/90f4c459e1bb71f79c2625f19d79753eb0fbfbd5/hoptimator-planner/src/main/java/com/linkedin/hoptimator/planner/PipelineRel.java#L95

[ryannedolan]

adding sqlDialect to pipeline

Makes sense to me!

[ryannedolan]

Thanks @HoganEdwardChu for #31

[HoganEdwardChu]

Thanks!!!!!

I am trying to add another sql time like spark. If there is any valuable progress I will pr! @ryannedolan