[SPARK-34607][SQL][2.4] Add `Utils.isMemberClass` to fix a malformed class name error on jdk8u #31747

maropu · 2021-03-05T00:27:49Z

What changes were proposed in this pull request?

This PR intends to fix a bug of objects.NewInstance if a user runs Spark on jdk8u and a given cls in NewInstance is a deeply-nested inner class, e.g.,.

  object OuterLevelWithVeryVeryVeryLongClassName1 {
    object OuterLevelWithVeryVeryVeryLongClassName2 {
      object OuterLevelWithVeryVeryVeryLongClassName3 {
        object OuterLevelWithVeryVeryVeryLongClassName4 {
          object OuterLevelWithVeryVeryVeryLongClassName5 {
            object OuterLevelWithVeryVeryVeryLongClassName6 {
              object OuterLevelWithVeryVeryVeryLongClassName7 {
                object OuterLevelWithVeryVeryVeryLongClassName8 {
                  object OuterLevelWithVeryVeryVeryLongClassName9 {
                    object OuterLevelWithVeryVeryVeryLongClassName10 {
                      object OuterLevelWithVeryVeryVeryLongClassName11 {
                        object OuterLevelWithVeryVeryVeryLongClassName12 {
                          object OuterLevelWithVeryVeryVeryLongClassName13 {
                            object OuterLevelWithVeryVeryVeryLongClassName14 {
                              object OuterLevelWithVeryVeryVeryLongClassName15 {
                                object OuterLevelWithVeryVeryVeryLongClassName16 {
                                  object OuterLevelWithVeryVeryVeryLongClassName17 {
                                    object OuterLevelWithVeryVeryVeryLongClassName18 {
                                      object OuterLevelWithVeryVeryVeryLongClassName19 {
                                        object OuterLevelWithVeryVeryVeryLongClassName20 {
                                          case class MalformedNameExample2(x: Int)
                                        }}}}}}}}}}}}}}}}}}}}

The root cause that Kris (@rednaxelafx) investigated is as follows (Kudos to Kris);

The reason why the test case above is so convoluted is in the way Scala generates the class name for nested classes. In general, Scala generates a class name for a nested class by inserting the dollar-sign ( $ ) in between each level of class nesting. The problem is that this format can concatenate into a very long string that goes beyond certain limits, so Scala will change the class name format beyond certain length threshold.

For the example above, we can see that the first two levels of class nesting have class names that look like this:

org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassName1$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassName1$OuterLevelWithVeryVeryVeryLongClassName2$

If we leave out the fact that Scala uses a dollar-sign ( $ ) suffix for the class name of the companion object, OuterLevelWithVeryVeryVeryLongClassName1's full name is a prefix (substring) of OuterLevelWithVeryVeryVeryLongClassName2.

But if we keep going deeper into the levels of nesting, you'll find names that look like:

org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$2a1321b953c615695d7442b2adb1$$$$ryVeryLongClassName8$OuterLevelWithVeryVeryVeryLongClassName9$OuterLevelWithVeryVeryVeryLongClassName10$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$2a1321b953c615695d7442b2adb1$$$$ryVeryLongClassName8$OuterLevelWithVeryVeryVeryLongClassName9$OuterLevelWithVeryVeryVeryLongClassName10$OuterLevelWithVeryVeryVeryLongClassName11$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$85f068777e7ecf112afcbe997d461b$$$$VeryLongClassName11$OuterLevelWithVeryVeryVeryLongClassName12$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$85f068777e7ecf112afcbe997d461b$$$$VeryLongClassName11$OuterLevelWithVeryVeryVeryLongClassName12$OuterLevelWithVeryVeryVeryLongClassName13$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$85f068777e7ecf112afcbe997d461b$$$$VeryLongClassName11$OuterLevelWithVeryVeryVeryLongClassName12$OuterLevelWithVeryVeryVeryLongClassName13$OuterLevelWithVeryVeryVeryLongClassName14$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$5f7ad51804cb1be53938ea804699fa$$$$VeryLongClassName14$OuterLevelWithVeryVeryVeryLongClassName15$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$5f7ad51804cb1be53938ea804699fa$$$$VeryLongClassName14$OuterLevelWithVeryVeryVeryLongClassName15$OuterLevelWithVeryVeryVeryLongClassName16$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$5f7ad51804cb1be53938ea804699fa$$$$VeryLongClassName14$OuterLevelWithVeryVeryVeryLongClassName15$OuterLevelWithVeryVeryVeryLongClassName16$OuterLevelWithVeryVeryVeryLongClassName17$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$69b54f16b1965a31e88968df1a58d8$$$$VeryLongClassName17$OuterLevelWithVeryVeryVeryLongClassName18$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$69b54f16b1965a31e88968df1a58d8$$$$VeryLongClassName17$OuterLevelWithVeryVeryVeryLongClassName18$OuterLevelWithVeryVeryVeryLongClassName19$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$69b54f16b1965a31e88968df1a58d8$$$$VeryLongClassName17$OuterLevelWithVeryVeryVeryLongClassName18$OuterLevelWithVeryVeryVeryLongClassName19$OuterLevelWithVeryVeryVeryLongClassName20$

with a hash code in the middle and various levels of nesting omitted.

The java.lang.Class.isMemberClass method is implemented in JDK8u as:
http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/tip/src/share/classes/java/lang/Class.java#l1425

    /**
     * Returns {@code true} if and only if the underlying class
     * is a member class.
     *
     * @return {@code true} if and only if this class is a member class.
     * @since 1.5
     */
    public boolean isMemberClass() {
        return getSimpleBinaryName() != null && !isLocalOrAnonymousClass();
    }

    /**
     * Returns the "simple binary name" of the underlying class, i.e.,
     * the binary name without the leading enclosing class name.
     * Returns {@code null} if the underlying class is a top level
     * class.
     */
    private String getSimpleBinaryName() {
        Class<?> enclosingClass = getEnclosingClass();
        if (enclosingClass == null) // top level class
            return null;
        // Otherwise, strip the enclosing class' name
        try {
            return getName().substring(enclosingClass.getName().length());
        } catch (IndexOutOfBoundsException ex) {
            throw new InternalError("Malformed class name", ex);
        }
    }

and the problematic code is getName().substring(enclosingClass.getName().length()) -- if a class's enclosing class's full name is longer than the nested class's full name, this logic would end up going out of bounds.

The bug has been fixed in JDK9 by https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8057919 , but still exists in the latest JDK8u release. So from the Spark side we'd need to do something to avoid hitting this problem.

This is the backport of #31733.

Why are the changes needed?

Bugfix on jdk8u.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Added tests.

maropu · 2021-03-05T01:39:04Z

It seems the test failed and I'll check it.

SparkQA · 2021-03-05T01:46:05Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40364/

SparkQA · 2021-03-05T02:03:38Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40364/

maropu · 2021-03-05T02:31:12Z

See #31745 (comment) for the failure reason.

SparkQA · 2021-03-05T02:50:40Z

Test build #135782 has finished for PR 31747 at commit d0d049c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
* inner class (See SPARK-34607 for details). This issue has already been fixed in jdk9+, so
//getSimpleBinaryName() returns null if a given class is a top-level class,

dongjoon-hyun · 2021-03-05T07:11:22Z

Could you make a PR to branch-2.4, @rednaxelafx ? Although we know that will fails, we can merge this PR and the original PR together.

rednaxelafx · 2021-03-05T09:04:41Z

Let me briefly describe the situation in a new comment in my original PR: #31709 (comment)

We definitely still want the isMemberClass fix in @maropu san's PR, since it avoids a Java InternalError that Spark SQL didn't know how to handle. But the test case needs some massaging to not fail the test, c.f. my comment linked above.

dongjoon-hyun · 2021-03-10T20:15:22Z

I made a relevant test suite fix for this.

[SPARK-34696][SQL][TESTS] Fix CodegenInterpretedPlanTest to generate correct test cases #31766

srowen · 2021-03-11T14:50:29Z

Should this not go into master + 3.x as well? or is it already fixed there?

dongjoon-hyun · 2021-03-11T15:47:33Z

@srowen . This is a backport of #31733 .
This has been blocked due to the UT failures of the relevant prior patches and the test suite bug ( #31766 ).

srowen · 2021-03-12T00:00:11Z

Jenkins retest this please

srowen · 2021-03-12T14:05:46Z

Jenkins retest this please

maropu · 2021-03-13T05:37:39Z

Is Jenkins unstable?

maropu · 2021-03-13T05:37:45Z

retest this please

SparkQA · 2021-03-13T05:58:44Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40608/

SparkQA · 2021-03-13T06:13:41Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40608/

viirya · 2021-03-15T01:12:34Z

I suppose this should be in 2.4.8, right? @maropu

viirya · 2021-03-15T19:13:25Z

retest this please

viirya · 2021-03-15T19:59:22Z

There is a test error in the GA:

[info]   Cause: java.lang.StringIndexOutOfBoundsException: String index out of range: -83
[info]   at java.lang.String.substring(String.java:1931)
[info]   at java.lang.Class.getSimpleBinaryName(Class.java:1448)
[info]   at java.lang.Class.getSimpleName(Class.java:1309)
[info]   at org.apache.spark.sql.catalyst.expressions.objects.NewInstance$$anonfun$11.apply(objects.scala:490)
...

maropu · 2021-03-16T00:27:06Z

I suppose this should be in 2.4.8, right? @maropu

Yea, I think so. I'll check the test failure.

maropu · 2021-03-16T04:29:49Z

I noticed that branch-2.4 does not have an interpreted implementation of SafeProjection, so the fallback cannot happen in some code path. For example, ExpressionEncoder directly uses GenerateSafeProjection (see the code below), so it throws an exception no matter what value we set to spark.sql.codegen.factoryMode.

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala

Line 271 in 3c627ad

    
           private lazy val constructProjection = GenerateSafeProjection.generate(deserializer :: Nil)

I think we have two options;

give up this PR and [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode #31709 in branch-2.4
merge this PR and [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode #31709 into branch-2.4 for changing the thrown exception from JVM StringIndexOutOfBoundsException into a Janino compilation error

I feel there is no much difference between them for users though, the second one looks better cuz StringIndexOutOfBoundsException gives users no error context, I think. WDHY? @viirya @dongjoon-hyun @HyukjinKwon

maropu · 2021-03-16T04:33:30Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala

@@ -487,7 +487,7 @@ case class NewInstance(
    ev.isNull = resultIsNull

    val constructorCall = outer.map { gen =>
-      s"${gen.value}.new ${cls.getSimpleName}($argString)"
+      s"${gen.value}.new ${Utils.getSimpleName(cls)}($argString)"


I temporarily apply this fix (#31709) to check if the tests can pass.

viirya · 2021-03-16T18:16:51Z

SQL tests seem all passed in GA? I only saw unrelated R install error. Triggered GA again.

viirya · 2021-03-16T18:17:05Z

retest this please

maropu · 2021-03-16T23:00:29Z

Yea, in the latest commit, I modified the test itself and applied the #31709 fix so that the GA tests can pass. The modified test (checkCompilationError) just checks that the compilation fails with codegen enabled/disabled. If we merge this fix into branch-2.4, I think we need to apply the #31709 fix first, and then merge this in.

viirya · 2021-03-18T01:20:54Z

I think we have two options;
* give up this PR and #31709 in branch-2.4

* merge this PR and #31709 into branch-2.4 for changing the thrown exception from JVM `StringIndexOutOfBoundsException` into a Janino compilation error
I feel there is no much difference between them for users though, the second one looks better cuz StringIndexOutOfBoundsException gives users no error context, I think. WDHY? @viirya @dongjoon-hyun @HyukjinKwon

Hmm, so even with this and #31709, there is still compilation error? Do we know what the compilation error is?

maropu · 2021-03-18T23:31:48Z

Hmm, so even with this and #31709, there is still compilation error? Do we know what the compilation error is?

Yea, right. All the branches have the same compilation error if branch-2.4 has #31709 and this PR. The only difference is that master/branch-3.1/branch-3.2 can fall back into the interpreted one, but branch-2.4 does not.

viirya · 2021-03-18T23:36:18Z

Yea, right. All the branches have the same compilation error if branch-2.4 has #31709 and this PR. The only difference is that master/branch-3.1/branch-3.2 can fall back into the interpreted one, but branch-2.4 does not.

Okay, then a compilation error instead of mystical StringIndexOutOfBoundsException seems better.

viirya · 2021-03-18T23:37:28Z

I'm fine to have this and #31709 to branch-2.4.

maropu · 2021-03-18T23:39:54Z

Thanks for the check, @viirya ! okay, I'll open a new PR to backport #31709 first.

viirya · 2021-03-24T17:41:19Z

@maropu #31709 was merged. Can you sync up this? If no problem, I think we can merge this.

maropu · 2021-03-24T23:52:57Z

Sure.

maropu · 2021-03-25T00:07:00Z

done

viirya · 2021-03-25T00:10:29Z

retest this please

SparkQA · 2021-03-25T00:48:06Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41068/

SparkQA · 2021-03-25T01:04:44Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41068/

SparkQA · 2021-03-25T03:00:36Z

Test build #136484 has finished for PR 31747 at commit a9c3188.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya

lgtm

viirya · 2021-03-25T06:55:50Z

Note that this is backport PR.

I will merge this to branch-2.4 tomorrow if no more comments.

cc @dongjoon-hyun @rednaxelafx @cloud-fan

viirya · 2021-03-26T02:19:33Z

Thanks all. Merging to branch-2.4.

…class name error on jdk8u ### What changes were proposed in this pull request? This PR intends to fix a bug of `objects.NewInstance` if a user runs Spark on jdk8u and a given `cls` in `NewInstance` is a deeply-nested inner class, e.g.,. ``` object OuterLevelWithVeryVeryVeryLongClassName1 { object OuterLevelWithVeryVeryVeryLongClassName2 { object OuterLevelWithVeryVeryVeryLongClassName3 { object OuterLevelWithVeryVeryVeryLongClassName4 { object OuterLevelWithVeryVeryVeryLongClassName5 { object OuterLevelWithVeryVeryVeryLongClassName6 { object OuterLevelWithVeryVeryVeryLongClassName7 { object OuterLevelWithVeryVeryVeryLongClassName8 { object OuterLevelWithVeryVeryVeryLongClassName9 { object OuterLevelWithVeryVeryVeryLongClassName10 { object OuterLevelWithVeryVeryVeryLongClassName11 { object OuterLevelWithVeryVeryVeryLongClassName12 { object OuterLevelWithVeryVeryVeryLongClassName13 { object OuterLevelWithVeryVeryVeryLongClassName14 { object OuterLevelWithVeryVeryVeryLongClassName15 { object OuterLevelWithVeryVeryVeryLongClassName16 { object OuterLevelWithVeryVeryVeryLongClassName17 { object OuterLevelWithVeryVeryVeryLongClassName18 { object OuterLevelWithVeryVeryVeryLongClassName19 { object OuterLevelWithVeryVeryVeryLongClassName20 { case class MalformedNameExample2(x: Int) }}}}}}}}}}}}}}}}}}}} ``` The root cause that Kris (rednaxelafx) investigated is as follows (Kudos to Kris); The reason why the test case above is so convoluted is in the way Scala generates the class name for nested classes. In general, Scala generates a class name for a nested class by inserting the dollar-sign ( `$` ) in between each level of class nesting. The problem is that this format can concatenate into a very long string that goes beyond certain limits, so Scala will change the class name format beyond certain length threshold. For the example above, we can see that the first two levels of class nesting have class names that look like this: ``` org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassName1$ org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassName1$OuterLevelWithVeryVeryVeryLongClassName2$ ``` If we leave out the fact that Scala uses a dollar-sign ( `$` ) suffix for the class name of the companion object, `OuterLevelWithVeryVeryVeryLongClassName1`'s full name is a prefix (substring) of `OuterLevelWithVeryVeryVeryLongClassName2`. But if we keep going deeper into the levels of nesting, you'll find names that look like: ``` org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$2a1321b953c615695d7442b2adb1$$$$ryVeryLongClassName8$OuterLevelWithVeryVeryVeryLongClassName9$OuterLevelWithVeryVeryVeryLongClassName10$ org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$2a1321b953c615695d7442b2adb1$$$$ryVeryLongClassName8$OuterLevelWithVeryVeryVeryLongClassName9$OuterLevelWithVeryVeryVeryLongClassName10$OuterLevelWithVeryVeryVeryLongClassName11$ org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$85f068777e7ecf112afcbe997d461b$$$$VeryLongClassName11$OuterLevelWithVeryVeryVeryLongClassName12$ org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$85f068777e7ecf112afcbe997d461b$$$$VeryLongClassName11$OuterLevelWithVeryVeryVeryLongClassName12$OuterLevelWithVeryVeryVeryLongClassName13$ org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$85f068777e7ecf112afcbe997d461b$$$$VeryLongClassName11$OuterLevelWithVeryVeryVeryLongClassName12$OuterLevelWithVeryVeryVeryLongClassName13$OuterLevelWithVeryVeryVeryLongClassName14$ org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$5f7ad51804cb1be53938ea804699fa$$$$VeryLongClassName14$OuterLevelWithVeryVeryVeryLongClassName15$ org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$5f7ad51804cb1be53938ea804699fa$$$$VeryLongClassName14$OuterLevelWithVeryVeryVeryLongClassName15$OuterLevelWithVeryVeryVeryLongClassName16$ org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$5f7ad51804cb1be53938ea804699fa$$$$VeryLongClassName14$OuterLevelWithVeryVeryVeryLongClassName15$OuterLevelWithVeryVeryVeryLongClassName16$OuterLevelWithVeryVeryVeryLongClassName17$ org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$69b54f16b1965a31e88968df1a58d8$$$$VeryLongClassName17$OuterLevelWithVeryVeryVeryLongClassName18$ org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$69b54f16b1965a31e88968df1a58d8$$$$VeryLongClassName17$OuterLevelWithVeryVeryVeryLongClassName18$OuterLevelWithVeryVeryVeryLongClassName19$ org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$69b54f16b1965a31e88968df1a58d8$$$$VeryLongClassName17$OuterLevelWithVeryVeryVeryLongClassName18$OuterLevelWithVeryVeryVeryLongClassName19$OuterLevelWithVeryVeryVeryLongClassName20$ ``` with a hash code in the middle and various levels of nesting omitted. The `java.lang.Class.isMemberClass` method is implemented in JDK8u as: http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/tip/src/share/classes/java/lang/Class.java#l1425 ``` /** * Returns {code true} if and only if the underlying class * is a member class. * * return {code true} if and only if this class is a member class. * since 1.5 */ public boolean isMemberClass() { return getSimpleBinaryName() != null && !isLocalOrAnonymousClass(); } /** * Returns the "simple binary name" of the underlying class, i.e., * the binary name without the leading enclosing class name. * Returns {code null} if the underlying class is a top level * class. */ private String getSimpleBinaryName() { Class<?> enclosingClass = getEnclosingClass(); if (enclosingClass == null) // top level class return null; // Otherwise, strip the enclosing class' name try { return getName().substring(enclosingClass.getName().length()); } catch (IndexOutOfBoundsException ex) { throw new InternalError("Malformed class name", ex); } } ``` and the problematic code is `getName().substring(enclosingClass.getName().length())` -- if a class's enclosing class's full name is *longer* than the nested class's full name, this logic would end up going out of bounds. The bug has been fixed in JDK9 by https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8057919 , but still exists in the latest JDK8u release. So from the Spark side we'd need to do something to avoid hitting this problem. This is the backport of #31733. ### Why are the changes needed? Bugfix on jdk8u. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added tests. Closes #31747 from maropu/SPARK34607-BRANCH2.4. Authored-by: Takeshi Yamamuro <yamamuro@apache.org> Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>

maropu · 2021-03-26T02:21:01Z

Thanks for the reviews, @viirya @cloud-fan !

viirya approved these changes Mar 5, 2021

View reviewed changes

maropu commented Mar 16, 2021

View reviewed changes

maropu changed the title ~~[SPARK-34607][SQL][2.4] Add Utils.isMemberClass to fix a malformed class name error on jdk8u~~ [WIP][SPARK-34607][SQL][2.4] Add Utils.isMemberClass to fix a malformed class name error on jdk8u Mar 19, 2021

maropu mentioned this pull request Mar 19, 2021

[SPARK-34596][SQL][2.4] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode #31888

Closed

maropu added 2 commits March 25, 2021 08:59

Fix

d7103ec

Fix

a9c3188

maropu force-pushed the SPARK34607-BRANCH2.4 branch from bd7dbf0 to a9c3188 Compare March 25, 2021 00:06

maropu changed the title ~~[WIP][SPARK-34607][SQL][2.4] Add Utils.isMemberClass to fix a malformed class name error on jdk8u~~ [SPARK-34607][SQL][2.4] Add Utils.isMemberClass to fix a malformed class name error on jdk8u Mar 25, 2021

viirya approved these changes Mar 25, 2021

View reviewed changes

cloud-fan approved these changes Mar 25, 2021

View reviewed changes

viirya closed this Mar 26, 2021

[SPARK-34607][SQL][2.4] Add Utils.isMemberClass to fix a malformed class name error on jdk8u #31747

[SPARK-34607][SQL][2.4] Add Utils.isMemberClass to fix a malformed class name error on jdk8u #31747

Conversation

maropu commented Mar 5, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

maropu commented Mar 5, 2021

SparkQA commented Mar 5, 2021

SparkQA commented Mar 5, 2021

maropu commented Mar 5, 2021

SparkQA commented Mar 5, 2021

dongjoon-hyun commented Mar 5, 2021

rednaxelafx commented Mar 5, 2021 • edited Loading

dongjoon-hyun commented Mar 10, 2021

srowen commented Mar 11, 2021

dongjoon-hyun commented Mar 11, 2021

srowen commented Mar 12, 2021

srowen commented Mar 12, 2021

maropu commented Mar 13, 2021

maropu commented Mar 13, 2021

SparkQA commented Mar 13, 2021

SparkQA commented Mar 13, 2021

viirya commented Mar 15, 2021

viirya commented Mar 15, 2021

viirya commented Mar 15, 2021

maropu commented Mar 16, 2021

maropu commented Mar 16, 2021

maropu Mar 16, 2021

Choose a reason for hiding this comment

viirya commented Mar 16, 2021

viirya commented Mar 16, 2021

maropu commented Mar 16, 2021

viirya commented Mar 18, 2021

maropu commented Mar 18, 2021

viirya commented Mar 18, 2021

viirya commented Mar 18, 2021

maropu commented Mar 18, 2021

viirya commented Mar 24, 2021

maropu commented Mar 24, 2021

maropu commented Mar 25, 2021

viirya commented Mar 25, 2021

SparkQA commented Mar 25, 2021

SparkQA commented Mar 25, 2021

SparkQA commented Mar 25, 2021

viirya left a comment

Choose a reason for hiding this comment

viirya commented Mar 25, 2021

viirya commented Mar 26, 2021

maropu commented Mar 26, 2021

[SPARK-34607][SQL][2.4] Add `Utils.isMemberClass` to fix a malformed class name error on jdk8u #31747

[SPARK-34607][SQL][2.4] Add `Utils.isMemberClass` to fix a malformed class name error on jdk8u #31747

rednaxelafx commented Mar 5, 2021 •

edited

Loading