[Feature][Transform-V2][SQL] Support case when clause for SQL Transform plugin (#5013) by javalover123 · Pull Request #5014 · apache/seatunnel

javalover123 · 2023-07-04T01:36:03Z

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

EricJoy2048 · 2023-07-11T08:25:16Z

Please add e2e for this update.

EricJoy2048

Please add document and test case for this pr.

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

javalover123 · 2023-07-11T14:13:18Z

OK, added. apache/seatunnel-website#253

EricJoy2048 · 2023-07-13T05:42:07Z

OK, added. apache/seatunnel-website#253

I am sorry, I didn't express myself clearly. You need update the document https://github.com/apache/seatunnel/blob/dev/docs/en/transform-v2/sql-functions.md

EricJoy2048 · 2023-07-13T05:43:29Z

Hi, @rewerma , Can you help to review this pr?

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

wu-a-ge · 2023-07-27T07:53:00Z

there's a bug,If it is bigint, the case when cannot be matched

wu-a-ge · 2023-07-27T07:53:28Z

@javalover123 there's a bug

wu-a-ge · 2023-07-27T07:56:26Z

besides int,tinyint smallint bigint can't nornal match

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

javalover123 · 2023-07-27T14:09:44Z

@wu-a-ge Updated, but I haven't run e2e test already.
5f5f623

wu-a-ge · 2023-07-30T15:41:06Z

@wu-a-ge Updated, but I haven't run e2e test already. 5f5f623

another bug,my job config :



env {
  # You can set engine configuration here
  execution.parallelism = 1
  job.mode = "BATCH"
  checkpoint.interval = 5000
  #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
}

source {
  # This is a example source plugin **only for test and demonstrate the feature source plugin**
  FakeSource {
    result_table_name = "fake"
    parallelism = 1
    schema = {
      fields {
        name = "string"
        age = "bigint"
      }
    }
    rows = [
      {fields = ["javalover123",6], kind = INSERT}
    ]
  }
}

transform {
  sql {
    source_table_name="fake"
    result_table_name="result_0"
    query="select name,case age when 6 then 1 else age end as age from fake"
  }
}

sink {
  jdbc {
    url = "jdbc:mysql://127.0.0.1:3306/seatunnel"
    driver = "com.mysql.cj.jdbc.Driver"
    user = "root"
    password = "123456"
    source_table_name="result_0"
    database="seatunnel"
    table="test"
    generate_sink_sql="true"
  }
}

exception:

2023-07-30 23:38:38,172 WARN  org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-709778] [5.1] Exception in org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask@64ec1e6
java.lang.RuntimeException: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
	at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:209) ~[classes/:?]
	at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:59) ~[classes/:?]
	at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:39) ~[classes/:?]
	at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:27) ~[classes/:?]
	at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.handleRecord(IntermediateBlockingQueue.java:71) ~[classes/:?]
	at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.collect(IntermediateBlockingQueue.java:51) ~[classes/:?]
	at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.collect(IntermediateQueueFlowLifeCycle.java:52) ~[classes/:?]
	at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.collect(TransformSeaTunnelTask.java:73) ~[classes/:?]
	at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:165) ~[classes/:?]
	at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.call(TransformSeaTunnelTask.java:78) ~[classes/:?]
	at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:613) [classes/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_281]
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) [?:1.8.0_281]
	at java.util.concurrent.FutureTask.run(FutureTask.java) [?:1.8.0_281]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_281]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_281]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]
Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
	at org.apache.seatunnel.connectors.seatunnel.jdbc.internal.converter.AbstractJdbcRowConverter.toExternal(AbstractJdbcRowConverter.java:145) ~[classes/:?]
	at org.apache.seatunnel.connectors.seatunnel.jdbc.internal.executor.SimpleBatchStatementExecutor.addToBatch(SimpleBatchStatementExecutor.java:45) ~[classes/:?]
	at org.apache.seatunnel.connectors.seatunnel.jdbc.internal.executor.SimpleBatchStatementExecutor.addToBatch(SimpleBatchStatementExecutor.java:31) ~[classes/:?]
	at org.apache.seatunnel.connectors.seatunnel.jdbc.internal.executor.BufferedBatchStatementExecutor.executeBatch(BufferedBatchStatementExecutor.java:51) ~[classes/:?]
	at org.apache.seatunnel.connectors.seatunnel.jdbc.internal.JdbcOutputFormat.attemptFlush(JdbcOutputFormat.java:204) ~[classes/:?]
	at org.apache.seatunnel.connectors.seatunnel.jdbc.internal.JdbcOutputFormat.flush(JdbcOutputFormat.java:169) ~[classes/:?]
	at org.apache.seatunnel.connectors.seatunnel.jdbc.sink.JdbcSinkWriter.prepareCommit(JdbcSinkWriter.java:88) ~[classes/:?]
	at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:159) ~[classes/:?]
	... 16 more

@javalover123

javalover123 · 2023-07-31T05:17:05Z

Thanks, can you give some help/advice? @wu-a-ge

wu-a-ge · 2023-07-31T05:58:58Z

Thanks, can you give some help/advice? @wu-a-ge

Thanks for your contribution. My current workaround is to directly adjust the toExternal method in the AbstractJdbcRowConverter class. All casts are parsed using the parse method that wraps the type, but I personally believe that it is still necessary to determine the data type to solve the problem of different types of conversions

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

javalover123 · 2023-08-01T01:49:38Z

Update, please check, thanks. @wu-a-ge
In another way, Number.*Value() method performance should be better when run with JDK 9 or above.

                case TINYINT:
                    statement.setByte(
                            statementIndex, ((Number) row.getField(fieldIndex)).byteValue());
                    break;
                case SMALLINT:
                    statement.setShort(
                            statementIndex, ((Number) row.getField(fieldIndex)).shortValue());
                    break;
                case INT:
                    statement.setInt(
                            statementIndex, ((Number) row.getField(fieldIndex)).intValue());
                    break;
                case BIGINT:
                    statement.setLong(
                            statementIndex, ((Number) row.getField(fieldIndex)).longValue());
                    break;
                case FLOAT:
                    statement.setFloat(
                            statementIndex, ((Number) row.getField(fieldIndex)).floatValue());
                    break;
                case DOUBLE:
                    statement.setDouble(
                            statementIndex, ((Number) row.getField(fieldIndex)).doubleValue());

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

wu-a-ge · 2023-08-01T15:47:08Z

Update, please check, thanks. @wu-a-ge In another way, Number.*Value() method performance should be better when run with JDK 9 or above.

                case TINYINT:
                    statement.setByte(
                            statementIndex, ((Number) row.getField(fieldIndex)).byteValue());
                    break;
                case SMALLINT:
                    statement.setShort(
                            statementIndex, ((Number) row.getField(fieldIndex)).shortValue());
                    break;
                case INT:
                    statement.setInt(
                            statementIndex, ((Number) row.getField(fieldIndex)).intValue());
                    break;
                case BIGINT:
                    statement.setLong(
                            statementIndex, ((Number) row.getField(fieldIndex)).longValue());
                    break;
                case FLOAT:
                    statement.setFloat(
                            statementIndex, ((Number) row.getField(fieldIndex)).floatValue());
                    break;
                case DOUBLE:
                    statement.setDouble(
                            statementIndex, ((Number) row.getField(fieldIndex)).doubleValue());

Found a bug,

caseExpression.getElseExpression() may be null，so there will throw a NullPointException

@javalover123

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

javalover123 · 2023-08-01T16:08:07Z

Update, please check, thanks. @wu-a-ge

wu-a-ge · 2023-08-07T08:59:16Z

Update, please check, thanks. @wu-a-ge

Hello,One scenario is not supported

case when num3>2 and num='abc' then 7  end as tinyint1

case when is a complex logical expression

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

javalover123 · 2023-08-08T02:15:29Z

Update, please check, thanks. @wu-a-ge

TyrantLucifer · 2023-08-11T10:17:00Z

@rewerma PTAL.

...nel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/ZetaSQLFunction.java

seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/ZetaSQLType.java

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

javalover123 · 2023-08-16T13:22:02Z

Update, please check, thanks. @rewerma

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

EricJoy2048 · 2023-09-11T06:11:09Z

It seems that different data types may have Bug that cannot be covered by test cases.
Can you add more e2e case to cover them? Hi, @wu-a-ge , do you have any suggestions ?

zhilinli123 · 2023-09-12T02:02:31Z

I don't know what test cases are missing, personal advice,I think more type tests can be added to make sure it passes，For example, string,int,double,

SELECT 
  column_name,
  CASE 
    WHEN column_name IN (value_1, value_2, ...) THEN result_1
    ...
    ELSE test_in_int,
   CASE 
    WHEN column_name IN ('value_1','value_2', ...) THEN result_1
    ...
    ELSE test_in_string
    CASE 
    WHEN (column_name IN ('value_1','value_2', ...)  or column_name!='value3' or  or column_int!=10)  THEN result_1
    ...
    ELSE test_in_or_string,
    CASE 
    WHEN column_name IN (value_1, value_2 ) THEN value_2 
    ELSE test_in_return
  END AS output_column
FROM table_name;

javalover123 · 2023-09-14T05:21:42Z

@EricJoy2048 @zhilinli123 I may add this week

EricJoy2048 · 2023-10-18T03:06:24Z

@EricJoy2048 @zhilinli123 I may add this week

Hi, Are there any updates on this PR?

javalover123 · 2023-10-18T03:27:37Z

Sorry, I've modified one test, but don't have any more time to work on this recently.

zhilinli123 · 2023-11-27T02:40:59Z

I don't see any movement in this pr, can you assign it to me temporarily and I will find time to finish it in the near future
@EricJoy2048 cc @javalover123

javalover123 · 2023-11-27T15:39:52Z

I don't see any movement in this pr, can you assign it to me temporarily and I will find time to finish it in the near future @EricJoy2048 cc @javalover123

@zhilinli123 Sorry, I can't see assign button, @EricJoy2048 please try.

EricJoy2048 · 2023-12-08T02:42:29Z

Ok, I will close this pr.

[Feature][Transform-V2][SQL] Support case when clause for SQL Transfo…

5a34420

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

EricJoy2048 previously approved these changes Jul 6, 2023

View reviewed changes

github-actions bot added approved reviewed labels Jul 6, 2023

davidzollo added the First-time contributor First-time contributor label Jul 9, 2023

EricJoy2048 reviewed Jul 11, 2023

View reviewed changes

javalover123 added 2 commits July 11, 2023 21:20

Merge branch 'apache:dev' into dev

617f3c9

[Feature][Transform-V2][SQL] Support case when clause for SQL Transfo…

58fcf39

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

javalover123 dismissed EricJoy2048’s stale review via 58fcf39 July 11, 2023 13:23

github-actions bot removed approved reviewed labels Jul 11, 2023

[Feature][Transform-V2][SQL] Support case when clause for SQL Transfo…

eae7045

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

javalover123 added 3 commits July 27, 2023 18:40

Merge branch 'apache:dev' into dev

e509faa

[Feature][Transform-V2][SQL] Support case when clause for SQL Transfo…

5f5f623

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

[Feature][Transform-V2][SQL] Support case when clause for SQL Transfo…

e5fd48c

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

javalover123 added 2 commits July 31, 2023 21:26

Merge branch 'apache:dev' into dev

eb7378c

[Feature][Transform-V2][SQL] Support case when clause for SQL Transfo…

495f2c8

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

[Feature][Transform-V2][SQL] Support case when clause for SQL Transfo…

c8c69aa

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

[Feature][Transform-V2][SQL] Support case when clause for SQL Transfo…

61c6e08

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

Merge branch 'apache:dev' into dev

bf4f519

[Feature][Transform-V2][SQL] Support case when clause for SQL Transfo…

45f11fc

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

rewerma reviewed Aug 14, 2023

View reviewed changes

...nel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/ZetaSQLFunction.java Outdated Show resolved Hide resolved

seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/ZetaSQLType.java Show resolved Hide resolved

[Feature][Transform-V2][SQL] Support case when clause for SQL Transfo…

a9f7383

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

[Feature][Transform-V2][SQL] Support case when clause for SQL Transfo…

f089f36

…rm plugin (apache#5013) Co-authored-by: javalover123 <javalover123@foxmail.com>

EricJoy2048 closed this Dec 8, 2023

Comments

Conversation

javalover123 commented Jul 4, 2023

Purpose of this pull request

Check list

Uh oh!

EricJoy2048 commented Jul 11, 2023

Uh oh!

EricJoy2048 left a comment

Choose a reason for hiding this comment

Uh oh!

javalover123 commented Jul 11, 2023

Uh oh!

EricJoy2048 commented Jul 13, 2023

Uh oh!

EricJoy2048 commented Jul 13, 2023

Uh oh!

wu-a-ge commented Jul 27, 2023

Uh oh!

wu-a-ge commented Jul 27, 2023

Uh oh!

wu-a-ge commented Jul 27, 2023

Uh oh!

javalover123 commented Jul 27, 2023

Uh oh!

wu-a-ge commented Jul 30, 2023

Uh oh!

javalover123 commented Jul 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wu-a-ge commented Jul 31, 2023

Uh oh!

javalover123 commented Aug 1, 2023

Uh oh!

wu-a-ge commented Aug 1, 2023

Uh oh!

javalover123 commented Aug 1, 2023

Uh oh!

wu-a-ge commented Aug 7, 2023

Uh oh!

javalover123 commented Aug 8, 2023

Uh oh!

TyrantLucifer commented Aug 11, 2023

Uh oh!

Uh oh!

Uh oh!

javalover123 commented Aug 16, 2023

Uh oh!

EricJoy2048 commented Sep 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhilinli123 commented Sep 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

javalover123 commented Sep 14, 2023

Uh oh!

EricJoy2048 commented Oct 18, 2023

Uh oh!

javalover123 commented Oct 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhilinli123 commented Nov 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

javalover123 commented Nov 27, 2023

Uh oh!

EricJoy2048 commented Dec 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

javalover123 commented Jul 31, 2023 •

edited

Loading

EricJoy2048 commented Sep 11, 2023 •

edited

Loading

zhilinli123 commented Sep 12, 2023 •

edited

Loading

javalover123 commented Oct 18, 2023 •

edited

Loading

zhilinli123 commented Nov 27, 2023 •

edited

Loading