[SPARK-54506][CORE] Add $init$ method to LogKey for backward compatibility

[gengliangwang]

What changes were proposed in this pull request?

Follow-up of https://issues.apache.org/jira/browse/SPARK-53064.

Add a static $init$ method to the LogKey interface for backward compatibility with libraries compiled against Spark 4.0 where LogKey was a Scala trait.

Why are the changes needed?

In Spark 4.0, LogKey was a Scala trait which generates a $init$ method during compilation. Libraries like Delta Lake compiled against Spark 4.0 expect this method to exist. Since LogKey has been changed to a Java interface, calling the $init$ method throws NoSuchMethodError:

java.lang.NoSuchMethodError: 'void org.apache.spark.internal.LogKey.$init$(org.apache.spark.internal.LogKey)'

This change adds an empty $init$ static method to maintain binary compatibility.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Added a new test LogKeySuite to verify the $init$ method exists and can be called without throwing an exception.

[gengliangwang]

cc @pan3793 as well

[dongjoon-hyun]

Thank you for pinging me, @gengliangwang .

cc @LuciferYang

[dongjoon-hyun]

Do you know why MIMA was unable to detect this LogKey API change, @gengliangwang ?

[dongjoon-hyun]

Ah, I found this is internal API, @gengliangwang . I don't think we need this PR, do we?

org.apache.spark.internal.LogKey

[gengliangwang]

I don't think we need this PR, do we?

@dongjoon-hyun It helps avoid potential backward-compatibility issues if external code extends the LogKey trait in Spark 4.0. It would be good to address this here.

[dongjoon-hyun]

I don't think we need this PR, do we?

@dongjoon-hyun It helps avoid potential backward-compatibility issues if external code extends the LogKey trait in Spark 4.0. It would be good to address this here.

+1 for your passion. However, I don't think that's the Apache Spark community API contract. Moreover, your claim is very misleading because the Scala version is also different between 4.0 and 4.1. I don't think we can claim that 3rd-party libraries compiled with Scala 2.13.16 can work with Spark 4.1 (on Scala 2.13.17) without any issue. Your report sounds like just a beginning of the all related issues.

Given that, let's not do this. They should recompile it against Apache Spark 4.1.0 in order to support Apache Spark 4.1.0.

[gengliangwang]

@dongjoon-hyun ok, I don't have strong opinion on this one.

[dongjoon-hyun]

Thank you for closing this PR~

[pan3793]

@gengliangwang, I think this is an expected behavior, SPARK-53064 only provides a feasible approach to allow developers to write a custom LogKey that keeps binary compatibility with Spark 4.0 and 4.1.

From my knowledge, downstream projects that heavily use the Spark internal API are hard to maintain binary compatibility across Spark feature versions. One example is that Iceberg used to do that in the Spark 3.0 and 3.1 period, and decided to abandon that after a short time.