Rework unification of Object and Any in Java/Scala interop #7966

retronym · 2019-04-04T05:01:32Z

References to j.l.Object from Java sources or classfiles use
definitions.ObjectTpeJava, which behaves like ObjectTpe but is
slackened to AnyTpe during =:= and <:< when this can help yield
positive results.

This subsumes two prior mechanisms (usages of obj2Any and
MethodType.isJava)

Fixes scala/bug#10418
Fixes scala/bug#11469

retronym · 2019-04-04T05:03:35Z

test/files/neg/abstract-report2.check

  def removeAll(x$1: java.util.Collection[_]): Boolean = ???
  def retainAll(x$1: java.util.Collection[_]): Boolean = ???
  def size(): Int = ???
-  def toArray[T](x$1: Array[T with Object]): Array[T with Object] = ???
+  def toArray[T <: Object](x$1: Array[T]): Array[T] = ???


Interesting change here. Needs to look at this more to decide if its a re- or pro-gression...

retronym · 2019-04-10T10:57:30Z

Chatted about this with @odersky and @adriaanm. A slightly cleaner way to express this might be to make ObjectTpeJava a type ref to a synthetic type alias (named 'Object', owned by java.lang, but not entered in its scope). The RHS of that alias would be a bounded wildcard type >: Object <: Any. We might need to make sure that the constraint solver deals with this type appropriately.

References to `j.l.Object` from Java sources or classfiles use `definitions.ObjectTpeJava`, which behaves like `ObjectTpe` but is slackened to `AnyTpe` during `=:=` and `<:<` when this can help yield positive results. This subsumes two prior mechanisms (usages of `obj2Any` and `MethodType.isJava`)

adriaanm · 2019-05-08T09:41:21Z

Dropping 81a7e77, adding another 11469 test to the first commit.

adriaanm · 2019-05-08T09:46:30Z

Running community build: https://scala-ci.typesafe.com/job/scala-2.13.x-integrate-community-build/2084/console (using compiler built from previous sha, but only diff is tests)

adriaanm · 2019-05-08T12:42:33Z

scala-js says /cc @sjrd

javalanglib/src/main/scala/java/lang/Iterable.scala:18: error: 
type arguments [T] do not conform to trait Iterator's type parameter bounds [E <: Object]
   def iterator(): Iterator[T]
                   ^

sjrd · 2019-05-08T12:57:20Z

That looks problematic. The file Iterable.scala is here:
https://github.com/scala-js/scala-js/blob/v1.0.0-M7/javalanglib/src/main/scala/java/lang/Iterable.scala
It literally contains

package java.lang

import java.util.Iterator

trait Iterable[T] {
  def iterator(): Iterator[T]
}

It doesn't seem like this pattern is particularly Scala.js-specific. It's a straightforward mention of a Java interface with a Scala type parameter.

adriaanm · 2019-05-08T12:59:22Z

So you don't do any weird tricks to rewire the info of the type param to use Object as its bound?

adriaanm · 2019-05-08T12:59:53Z

Sorry, scratch that. I thought I was looking at the source of Iterator

sjrd · 2019-05-08T13:03:12Z

So you don't do any weird tricks to rewire the info of the type param to use Object as its bound?

OMG no. I don't touch the typer. I don't even touch infos at all, unless in the presence of types that extend js.Any.

adriaanm · 2019-05-08T13:03:42Z

Haha ok phew 😅

Still weird, as this PR's standard scalac compiles this source file without problem.

sjrd · 2019-05-08T13:11:43Z

Looks like a dbuild-specific issue. I cannot reproduce the issue locally with

> set resolvers in Global += "scala-pr-validation" at "https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots/"
> ++2.13.0-pre-d273f13-SNAPSHOT
> javalanglib/compile

I hate dbuild-specific issues 😫

adriaanm · 2019-05-08T13:16:25Z

Bizarre. Dbuild is actually using 2.13.0-pre-751a34f-SNAPSHOT, which should be identical. Could you please try with that one as well?

sjrd · 2019-05-08T14:15:44Z

Bizarre. Dbuild is actually using 2.13.0-pre-751a34f-SNAPSHOT, which should be identical. Could you please try with that one as well?

Same outcome: works fine locally.

adriaanm · 2019-05-08T14:27:28Z

Thanks. Well, I guess I'm in for some fun dbuild debugging :-)

adriaanm · 2019-05-08T14:43:57Z

Well, at least it fixed the issue with jackson-module-scala -- it built fine with my workaround reverted

Scala sha after merging scala/scala#7966

adriaanm · 2019-05-09T15:07:39Z

[EDIT]: the akka testkit breakage was not due to this PR, scala/bug#11523

Seems like this did break something in akka testkit -- haven't looked beyond copy pasting the error from https://scala-ci.typesafe.com/job/scala-2.13.x-integrate-community-build/2087/consoleText:

akka-testkit/src/main/java/akka/testkit/JavaTestKit.java:150:1: <R>compose(scala.PartialFunction<R,T1>) in scala.runtime.AbstractPartialFunction cannot implement <R>compose(scala.PartialFunction<R,A>) in scala.PartialFunction
  return type scala.PartialFunction<R,R> is not compatible with scala.PartialFunction<R,java.lang.Object>
          new JavaPartialFunction<Object, Object>() {
akka-testkit/src/main/java/akka/testkit/JavaTestKit.java:561:1: <R>compose(scala.PartialFunction<R,T1>) in scala.runtime.AbstractPartialFunction cannot implement <R>compose(scala.PartialFunction<R,A>) in scala.PartialFunction
  return type scala.PartialFunction<R,R> is not compatible with scala.PartialFunction<R,T>
                  new CachingPartialFunction<Object, T>() {

(akka-testkit / Compile / compileIncremental) javac returned non-zero exit code

adriaanm · 2019-05-13T08:06:49Z

/cc @anatoliykmetyuk this would be a good candidate to forward port to dotc. Note the follow-up PR, which makes the special ObjectJavaTpe truly unique.

SethTisue · 2019-05-17T04:10:43Z

#8049 is a followup to this

ches · 2019-05-24T10:12:25Z

this would be a good candidate to forward port to dotc

@adriaanm Is this likely to get backported too? If it resolves jdk11 migration issues like scala/bug#10418 for 2.11/12.

SethTisue · 2019-08-24T21:38:31Z

as far as I know a backport is unlikely given the tricky nature of the changes made.

smarter · 2020-01-14T20:26:47Z

This PR description says:

References to j.l.Object from Java sources or classfiles use
definitions.ObjectTpeJava

But as far as I can tell, ObjectTpeJava is only used for references to Object coming from classfiles and never for references coming from .java source files, and in fact some of the tests in this PR that are only run under separate compilation currently fail under joint compilation:

% scalac test/files/pos/t11469_2/Test_1.java test/files/pos/t11469_2/Test_2.scala
test/files/pos/t11469_2/Test_2.scala:5: error: name clash between defined and inherited member:
def m(a: java.util.Optional[_]): Unit in class B and
override def m(a: java.util.Optional[_ <: Object]): Unit at line 5
have same type after erasure: (a: java.util.Optional)Unit
    override def m(a: java.util.Optional[_ <: Object]): Unit = { }
                 ^
one error found

Is this intentional ?

retronym · 2020-01-14T23:01:51Z

But as far as I can tell, ObjectTpeJava is only used for references to Object coming from classfiles and never for references coming from .java source files

The change to checkAccessible works for references to Object from .java source files. But maybe there is a hole in way we expand Foo[_] to Foo[_ <: Object]...

retronym · 2020-01-15T00:57:39Z

#8638

In Java unlike Scala, `Object` is the top type, this leads to various usability problems when attempting to call or override a Java method from Scala. So far, we relied mainly on one mechanism to improve the situation: in the ClassfileParser, some references to `Object` in signatures were replaced by `Any` (cf `objToAny`). But this had several shortcomings: - To compensate for this substitution, `TypeComparer#matchingMethodParams` had to special case Any in Java methods and treat it like Object - There were various situation were this substitution was not applied, notably when using varargs (`Object... args`) or when jointly compiling .java sources since this is handled by JavaParser and not ClassfileParser. This commit replaces all of this by a more systematic solution: all references to `Object` in Java definitions (both in classfiles and .java source files) are replaced by a special type `FromJavaObject` which is a type alias of `Object` with one extra subtyping rule: tp <:< FromJavaObject is equivalent to: tp <:< Any See the documentation of `FromJavaObjectSymbol` for details on why this makes sense. This solution is very much inspired by scala/scala#7966 (which was refined in scala/scala#8049 and scala/scala#8638) with two main differences: - We use a type with its own symbol and name distinct from `java.lang.Object`, because this type can show up in inferred types and therefore needs to be preserved when pickling so that unpickled trees pass `-Ycheck`. - Unlike in Scala 2, we cannot get rid of `JavaMethodType` because when calling `Type#signature` we need to know whether we're in a Java method or not, because signatures are based on erased types and erasure differs between Scala and Java (we cannot ignore this and always base signatures on the Scala erasure, because signatures are used to distinguish between overloads and overrides so they must agree with the actual JVM bytecode signature). Fixes scala#7467, scala#7963, scala#8588, scala#8977.

scala-jenkins added this to the 2.13.1 milestone Apr 4, 2019

retronym commented Apr 4, 2019

View reviewed changes

retronym requested a review from adriaanm April 4, 2019 06:05

retronym force-pushed the ticket/10418 branch 3 times, most recently from 3426455 to 565be77 Compare April 5, 2019 06:44

retronym force-pushed the ticket/10418 branch from 47bd07a to 333797c Compare April 10, 2019 15:35

adriaanm mentioned this pull request May 6, 2019

problem with erased generics in Java and overriding scala/bug#11469

Closed

retronym force-pushed the ticket/10418 branch 3 times, most recently from d83faca to 81a7e77 Compare May 7, 2019 11:11

adriaanm modified the milestones: 2.13.1, 2.13.0-RC2 May 8, 2019

adriaanm force-pushed the ticket/10418 branch from 81a7e77 to d273f13 Compare May 8, 2019 09:41

adriaanm changed the title ~~WIP Rework the way we unify Object and Any in Java/Scala interop~~ Rework unification of Object and Any in Java/Scala interop May 8, 2019

adriaanm added a commit to scala/community-build that referenced this pull request May 8, 2019

Bump to 2.13.0-pre-d73f85d

c7e53e5

Scala sha after merging scala/scala#7966

adriaanm added a commit that referenced this pull request May 9, 2019

Some comments as follow up for #7966

6fcc7eb

adriaanm mentioned this pull request May 9, 2019

investigate javac compilation error in akka testkit scala/bug#11523

Closed

adriaanm added a commit to adriaanm/scala-js that referenced this pull request May 10, 2019

repro for regression due to scala/scala#7966?

9bb932c

adriaanm mentioned this pull request May 10, 2019

ObjectTpeJava is truly unique; bounds; isAnyTpe #8049

Merged

adriaanm added the java interop label May 14, 2019

lrytz mentioned this pull request May 14, 2019

should consider generic sig and its erasure as overriding for Java member pairs scala/bug#11530

Closed

adriaanm mentioned this pull request May 15, 2019

Unable to use functional interface as a SAM target type scala/bug#11361

Closed

SethTisue added the release-notes worth highlighting in next release notes label May 17, 2019

SethTisue mentioned this pull request May 23, 2019

Update scala 2.13.0-M5 to 2.13.0-RC1 lightbend-labs/scala-logging#159

Closed

ekrich mentioned this pull request Jun 11, 2019

Java Object interopt changed in Scala 2.13.0 ekrich/sconfig#35

Open

adriaanm added the backport candidate label Jun 25, 2019

joroKr21 mentioned this pull request Aug 31, 2019

Remove the verbosity of a partest #8389

Merged

smarter mentioned this pull request Mar 22, 2020

java interop: not auto box scala/scala3#8588

Closed

smarter mentioned this pull request Jun 5, 2020

Dotty rejects all overrides of some Java methods as duplicate methods scala/scala3#7467

Closed

smarter mentioned this pull request Aug 19, 2020

Improve handling of references to Object coming from Java code scala/scala3#9601

Merged

retronym mentioned this pull request Oct 18, 2021

x retronym/scratch#1

Open

ronnnnnnnnnnnnn mentioned this pull request Nov 28, 2021

Investigate alternatives to caffeine cache key AnyRef type constraint cb372/scalacache#603

Open

retronym mentioned this pull request Dec 10, 2021

Correctly deal with Java/Scala impedence mismatch between Object/Any when using -Ypickle-write-java scala/bug#12512

Closed

som-snytt mentioned this pull request Apr 5, 2023

Impossible for Scala 2.12 to handle overloaded Java method that contains both varargs and parameters of same type scala/bug#12765

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rework unification of Object and Any in Java/Scala interop #7966

Rework unification of Object and Any in Java/Scala interop #7966

retronym commented Apr 4, 2019

retronym Apr 4, 2019 •

edited

retronym commented Apr 10, 2019

adriaanm commented May 8, 2019

adriaanm commented May 8, 2019

adriaanm commented May 8, 2019

sjrd commented May 8, 2019

adriaanm commented May 8, 2019

adriaanm commented May 8, 2019

sjrd commented May 8, 2019

adriaanm commented May 8, 2019

sjrd commented May 8, 2019

adriaanm commented May 8, 2019

sjrd commented May 8, 2019

adriaanm commented May 8, 2019

adriaanm commented May 8, 2019

adriaanm commented May 9, 2019 •

edited by lrytz

adriaanm commented May 13, 2019

SethTisue commented May 17, 2019

ches commented May 24, 2019

SethTisue commented Aug 24, 2019

smarter commented Jan 14, 2020

retronym commented Jan 14, 2020 •

edited

retronym commented Jan 15, 2020

Rework unification of Object and Any in Java/Scala interop #7966

Rework unification of Object and Any in Java/Scala interop #7966

Conversation

retronym commented Apr 4, 2019

retronym Apr 4, 2019 • edited

Choose a reason for hiding this comment

retronym commented Apr 10, 2019

adriaanm commented May 8, 2019

adriaanm commented May 8, 2019

adriaanm commented May 8, 2019

sjrd commented May 8, 2019

adriaanm commented May 8, 2019

adriaanm commented May 8, 2019

sjrd commented May 8, 2019

adriaanm commented May 8, 2019

sjrd commented May 8, 2019

adriaanm commented May 8, 2019

sjrd commented May 8, 2019

adriaanm commented May 8, 2019

adriaanm commented May 8, 2019

adriaanm commented May 9, 2019 • edited by lrytz

adriaanm commented May 13, 2019

SethTisue commented May 17, 2019

ches commented May 24, 2019

SethTisue commented Aug 24, 2019

smarter commented Jan 14, 2020

retronym commented Jan 14, 2020 • edited

retronym commented Jan 15, 2020

retronym Apr 4, 2019 •

edited

adriaanm commented May 9, 2019 •

edited by lrytz

retronym commented Jan 14, 2020 •

edited