[Bug] [Seatunnel Transform] FieldMapper是否支持对象.属性的映射关系

[oldwolf1984]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

Seatunnel 脚本 Source中有个data类型的对象，其中有个属性A，在transform fieldMapper中是否可以映射到目标表中的一个字段B中，还是说目前seatunnel中暂时不支持这种对象.属性的映射关系，还是说我的写法有问题。求助大神解答

SeaTunnel Version

2.3.5

SeaTunnel Config

无

Running Command

运行脚本

Error Exception

ErrorDescription:[The input fields 'data.MATNR' of 'FieldMapper' transform not found in upstream schema]

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[liunaijie]

you can use sql transform and inner query feature.
the query like select id, data.MANTER as new_col ...

[oldwolf1984]

我的源库和目标库都是MongoDB，也可以使用sql么？

[oldwolf1984]

使用sql 确实可以解决了，但是又有了一个新的问题

[liunaijie]

使用sql 确实可以解决了，但是又有了一个新的问题

when you use sql tranfrom.
the field_mapper will ignore. the transform result will same with you query result.

You can change to console sink to verify the result struct.

And you can use multiple tranfroms to generate the result you wanted,
like

transform {
  sql {
       query = ""
       result_table_name = "sql_res1"
   }

   sql {
      query = ""
       source_table_name = "sql_res1"
   }

}

[oldwolf1984]

你好 我按照上面发的写了两个sql标签，执行的时候后台有报错了
024-05-23 15:35:06,093 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.JobDefineCheckException: The source/transform(MongoDB) is not configured with 'result_table_name' option
at org.apache.seatunnel.engine.core.parse.ConfigParserUtil.checkExistTableId(ConfigParserUtil.java:237)
at org.apache.seatunnel.engine.core.parse.ConfigParserUtil.checkComplexGraph(ConfigParserUtil.java:155)
at org.apache.seatunnel.engine.core.parse.ConfigParserUtil.checkGraph(ConfigParserUtil.java:75)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:179)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:156)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:149)
... 2 more
Caused by: org.apache.seatunnel.api.configuration.util.OptionValidationException: ErrorCode:[API-02], ErrorDescription:[Option item validate failed] - The option("result_table_name") is incorrectly configured, please refer to the doc: When result_table_name is not specified, the data processed by this plugin will not be registered as a data set (dataStream/dataset) that can be directly accessed by other plugins, or called a temporary table (table)When result_table_name is specified, the data processed by this plugin will be registered as a data set (dataStream/dataset) that can be directly accessed by other plugins, or called a temporary table (table) . The data set (dataStream/dataset) registered here can be directly accessed by other plugins by specifying source_table_name .
... 9 more

[oldwolf1984]

[liunaijie]

1. the gived config is not complete
   it should be ( still ignore som config) :

source  {
   xxx {
       result_table_name = "source_result"
       ....
   }
}

transform {
  sql {
       query = ""
       source_table_name = "source_result"
       result_table_name = "sql_res1"    
   }

   sql {
      query = ""
       source_table_name = "sql_res1"
       result_table_name = "sql_res2"  // your exception is this config missing parameters
   }

}

sink {

   xxx {
      source_table_name = "sql_res2"
      ......
   }

}

2. in your second query, you want query from sink table, this is not right. you can only query from the source table.

data flow is source -> transform -> sink

and also not support query with multiple table using like join

3. for your case, your can try (i am not sure it can work, just suggestion)
   a. select data.A as data.B from source
   b. using json tranform

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.