Skip to content

Commit bd751df

Browse files
pan3793ulysses-you
authored andcommitted
[KYUUBI #2060] Clear job group for init SQL
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> We need to clear job group information for init SQL, otherwise in incremental collection mode, the fetchNext will always display wrong description. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate Before this patch: <img width="1916" alt="incremental_collect" src="https://user-images.githubusercontent.com/26535726/157165964-502ad809-9d99-44de-b08c-131d135ce33d.png"> After this patch: <img width="1917" alt="Xnip2022-03-08_12-04-50" src="https://user-images.githubusercontent.com/26535726/157165975-68eddf24-e0ad-4db2-931c-c75541802a7b.png"> - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2060 from pan3793/clear. Closes #2060 76a6737 [Cheng Pan] Clear job group for init SQL Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> (cherry picked from commit f8d9010) Signed-off-by: ulysses-you <ulyssesyou@apache.org>
1 parent 98e36de commit bd751df

File tree

2 files changed

+7
-1
lines changed

2 files changed

+7
-1
lines changed

externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,7 @@ object SparkSQLEngine extends Logging {
133133
interruptOnCancel = true)
134134
debug(s"Execute session initializing sql: $sqlStr")
135135
session.sql(sqlStr).isEmpty
136+
session.sparkContext.clearJobGroup()
136137
}
137138
session
138139
}

externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/session/SparkSQLSessionManager.scala

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,8 +66,13 @@ class SparkSQLSessionManager private (name: String, spark: SparkSession)
6666
} else {
6767
val ss = spark.newSession()
6868
this.conf.get(ENGINE_SESSION_INITIALIZE_SQL).foreach { sqlStr =>
69-
ss.sparkContext.setJobDescription(sqlStr)
69+
ss.sparkContext.setJobGroup(
70+
"engine_initializing_queries",
71+
sqlStr,
72+
interruptOnCancel = true)
73+
debug(s"Execute session initializing sql: $sqlStr")
7074
ss.sql(sqlStr).isEmpty
75+
ss.sparkContext.clearJobGroup()
7176
}
7277
ss
7378
}

0 commit comments

Comments
 (0)