[bug]Data service execution failed

[yuankang134]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Problem Description

The error is as follows:
(报错如下)

2022-03-09 22:20:04.020 INFO Variables substitution ended successfully

Job with jobId : IDE_janicegong_spark_16 and execID : IDE_janicegong_spark_16 submitted

2022-03-09 22:20:05.020 INFO You have submitted a new job, script code (after variable substitution) is

SCRIPT CODE

val executeCode = """select * from hduser05db.student limit 100

"""

val df = sqlContext.sql(executeCode)

val realMetaInfo = MetaDataInfoTool.getMetaDataInfo(sqlContext, executeCode, df.asInstanceOf[DataFrame])

if (realMetaInfo == """[hduser05db.student]"""){

show(df)

} else {

throw new Exception("库表名和发布时的不一致,不能进行执行")

}

SCRIPT CODE

2022-03-09 22:20:05.020 INFO Your job is accepted, jobID is exec_id018018linkis-cg-entrancebdphdp11ide01:9205IDE_janicegong_spark_16 and taskID is 1622207 in ServiceInstance(linkis-cg-entrance, bdphdp11ide01:9205). Please wait it to be scheduled

job is scheduled.

2022-03-09 22:20:05.020 INFO Your job is Scheduled. Please wait it to run.

Your job is being scheduled by orchestrator.

2022-03-09 22:20:05.020 INFO job is running.

2022-03-09 22:20:05.020 INFO Your job is Running now. Please wait it to complete.

2022-03-09 22:20:05.020 INFO Job with jobGroupId : 1622207 and subJobId : 1759811 was submitted to Orchestrator.

2022-03-09 22:20:05.020 INFO Background is starting a new engine for you,execId astJob_41_codeExec_41 mark id is mark_41, it may take several seconds, please wait

2022-03-09 22:21:22.021 INFO EngineConn local log path: ServiceInstance(linkis-cg-engineconn, bdpdws110004:24737) /data/bdp/linkis/janicegong/20220309/spark/c8537eca-254c-47ec-b3d2-9b9231153395/logs

2022-03-09 22:21:22.021 INFO yarn application id: application_1645708917744_19632

2022-03-09 22:21:22.021 INFO yarn application id: application_1645708917744_19632

2022-03-09 22:21:22.021 INFO yarn application id: application_1645708917744_19632

2022-03-09 22:21:22.021 INFO yarn application id: application_1645708917744_19632

scala> val executeCode = """select * from hduser05db.student limit 100

"""

2022-03-09 22:21:22.021 INFO yarn application id: application_1645708917744_19632

scala> val df = sqlContext.sql(executeCode)

2022-03-09 22:21:23.021 INFO yarn application id: application_1645708917744_19632

scala> val realMetaInfo = MetaDataInfoTool.getMetaDataInfo(sqlContext, executeCode, df.asInstanceOf[DataFrame])

2022-03-09 22:21:23.021 INFO yarn application id: application_1645708917744_19632

at show(:40)

... 82 elided

at scala.Option.getOrElse(Option.scala:121)

at scala.Option.getOrElse(Option.scala:121)

at scala.Option.getOrElse(Option.scala:121)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)

at scala.collection.immutable.List.foreach(List.scala:392)

at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)

at scala.collection.immutable.List.map(List.scala:296)

at scala.Option.getOrElse(Option.scala:121)

at scala.Option.getOrElse(Option.scala:121)

at scala.Option.getOrElse(Option.scala:121)

at scala.Option.getOrElse(Option.scala:121)

... 85 more

2022-03-09 22:21:25.283 ERROR [Linkis-Default-Scheduler-Thread-20] org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor 62 error - Execute code error for org.apache.linkis.engineplugin.spark.exception.SparkEngineException: errCode: 40002 ,desc: dataFrame to local exception ,ip: bdpdws110004 ,port: 24737 ,serviceKind: linkis-cg-engineconn

at show(:40)

... 82 elided

Description

No response

Use case

No response

solutions

No response

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[yuankang134]

this bug had fixed on dss-1.1.0