[SPARK-38510][SQL] Retry ClassSymbol.selfType to work around cyclic references #35852
Conversation
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveScalaReflectionSuite.scala
Outdated
Show resolved
Hide resolved
|
Can one of the admins verify this patch? |
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
Show resolved
Hide resolved
| @@ -639,7 +643,7 @@ object ScalaReflection extends ScalaReflection { | |||
| def getConstructorParameters(cls: Class[_]): Seq[(String, Type)] = { | |||
| val m = runtimeMirror(cls.getClassLoader) | |||
| val classSymbol = m.staticClass(cls.getName) | |||
| val t = classSymbol.selfType | |||
| val t = selfType(classSymbol) | |||
There was a problem hiding this comment.
Is this limited only to selfType ?
There was a problem hiding this comment.
Unfortunately, I do not have the technical knowhow to answer this question. What I can say is that ScalaReflection also uses ClassSymbol.companion which seems to work fine in this case. ClassSymbol.annotations fails, as might be expected, but it is not used anywhere in this class.
The issue itself is reproducible all the way back to 2.3 (at least). But I don't think this is too critical since |
|
Merged to master (3.4.0) |
…eferences Retry `ClassSymbol.selfType` in case we receive a cyclic reference exception. `ClassSymbol.selfType` fails on certain classes like `HiveGenericUDF` due to cyclic reference. This issue is due to [bug#12190 in Scala](scala/bug#12190) which does not handle cyclic references in Java annotations correctly. The cyclic reference in this case comes from `InterfaceAudience` annotation which [annotates itself](https://github.com/apache/hadoop/blob/db8ae4b65448c506c9234641b2c1f9b8e894dc18/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/InterfaceAudience.java#L45). This annotation class is present in the type hierarchy of `HiveGenericUDF`. To workaround the issue, we can just retry the operation and it will succeed on the retry probably because the annotation is partially resolved from the previous attempt. No Added a unit test. This unit test prefers to be run a separate JVM as the error for cyclic references may not be thrown if the annotation class is previously loaded by some other test and so may be dependent on test execution order. Closes apache#35852 from shardulm94/spark-38510. Authored-by: Shardul Mahadik <smahadik@linkedin.com> Signed-off-by: Sean Owen <srowen@gmail.com> (cherry picked from commit 91614ff) RB=3246777 BUG=LIHADOOP-63262 G=spark-reviewers R=tgudivad,ekrogen A=ekrogen RB=3456494 BUG=LIHADOOP-63952 G=spark-reviewers R=ekrogen,tgudivad A=ekrogen
What changes were proposed in this pull request?
Retry
ClassSymbol.selfTypein case we receive a cyclic reference exception.Why are the changes needed?
ClassSymbol.selfTypefails on certain classes likeHiveGenericUDFdue to cyclic reference. This issue is due to bug#12190 in Scala which does not handle cyclic references in Java annotations correctly. The cyclic reference in this case comes fromInterfaceAudienceannotation which annotates itself. This annotation class is present in the type hierarchy ofHiveGenericUDF.To workaround the issue, we can just retry the operation and it will succeed on the retry probably because the annotation is partially resolved from the previous attempt.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Added a unit test. This unit test prefers to be run a separate JVM as the error for cyclic references may not be thrown if the annotation class is previously loaded by some other test and so may be dependent on test execution order.