Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scalac 2.13.1 "incompatible type in overriding" subclassing Hadoop FileSystem that javac accepts #11840

Open
srowen opened this issue Dec 30, 2019 · 3 comments

Comments

@srowen
Copy link

@srowen srowen commented Dec 30, 2019

Hi all, long time listener, first time poster. I have an odd scalac 2.13.1 issue that's affecting Apache Spark's transition to 2.13 (https://issues.apache.org/jira/browse/SPARK-25075 -> https://issues.apache.org/jira/browse/SPARK-30132)

It occurs in scalac 2.13.1 but not 2.12.10. This occurs in Java 8. I didn't see an obvious open issue here, or easy answer from the gitter channel, but #11575 looks possibly related.

Consider this simple subclass of a Hadoop LocalFileSystem class:

object foo extends org.apache.hadoop.fs.LocalFileSystem {}

Compiled with scalac 2.13.1, I get:

$ scalac -cp ~/.m2/repository/org/apache/hadoop/hadoop-common/3.2.0/hadoop-common-3.2.0.jar foo.scala 
foo.scala:1: error: incompatible type in overriding
def appendFile(x$1: org.apache.hadoop.fs.Path): org.apache.hadoop.fs.FSDataOutputStreamBuilder[S,B] forSome { type S <: org.apache.hadoop.fs.FSDataOutputStream; type B <: org.apache.hadoop.fs.FSDataOutputStreamBuilder[S,B] } (defined in class FileSystem)
  with def appendFile(x$1: org.apache.hadoop.fs.Path): org.apache.hadoop.fs.FSDataOutputStreamBuilder (defined in class FilterFileSystem);
 found   : (x$1: org.apache.hadoop.fs.Path)org.apache.hadoop.fs.FSDataOutputStreamBuilder
 required: (x$1: org.apache.hadoop.fs.Path)org.apache.hadoop.fs.FSDataOutputStreamBuilder[S,B] forSome { type S <: org.apache.hadoop.fs.FSDataOutputStream; type B <: org.apache.hadoop.fs.FSDataOutputStreamBuilder[S,B] }
object foo extends org.apache.hadoop.fs.LocalFileSystem {}
       ^

This says that the compiled class hierarchy for this Hadoop class is invalid, though naturally, javac has accepted and compiled it. The definition in FileSystem.java in Hadoop 3.2.0:

  public FSDataOutputStreamBuilder appendFile(Path path) {
    return new FileSystemDataOutputStreamBuilder(this, path).append();
  }

The subtype FilterFileSystem.java has:

  public FSDataOutputStreamBuilder appendFile(Path path) {
    return new FileSystemDataOutputStreamBuilder(this, path).append();
  }

The signature is evidently identical. Now, FSDataOutputStreamBuilder does take two generic types, not specified here in the Hadoop source code. I kind of understand the error, that you can't say that the generic types in the overload are consistent.

But is this something that scalac should reject?

@Jasper-M

This comment has been minimized.

Copy link
Member

@Jasper-M Jasper-M commented Dec 30, 2019

Looks like it might be related to #11770

@joshlemer

This comment has been minimized.

Copy link
Member

@joshlemer joshlemer commented Dec 30, 2019

Minimized example:

Example.java

public class Example {
    public static abstract class X<A extends java.lang.Object, B extends X<A, B>> {}

    public static abstract class Y {
        public X appendFile(String s) { return null; }
    }

    public static class Z extends Y {
        @Override
        public X appendFile(String s) { return null; }
    }
}

Main.scala

object foo extends Example.Z {}

Error:

Error:(1, 8) incompatible type in overriding
def appendFile(s: String): Example.X[A,B] forSome { type A; type B <: Example.X[A,B] } (defined in class Y)
  with def appendFile(s: String): Example.X (defined in class Z);
 found   : (s: String)Example.X
 required: (s: String)Example.X[A,B] forSome { type A; type B <: Example.X[A,B] }
object foo extends Example.Z {}
@Jasper-M

This comment has been minimized.

Copy link
Member

@Jasper-M Jasper-M commented Dec 30, 2019

Probably also #11584

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.