Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix #1770: Fix __scala_== with ScalaNumbers. #1805

Merged
merged 3 commits into from May 25, 2020

Conversation

LeeTibbert
Copy link
Contributor

@LeeTibbert LeeTibbert commented May 16, 2020

  • This PR was motivated by Issue __scala_.== is borked with BigInt & BigInteger, almost certainly both BigDecimal #1770:
    "_scala== is borked with BigInt & BigInteger, both BigDecimal".

    That issue is now resolved.

  • This PR needs review from experts in Scala & Java equality. It affects a key and widely used
    piece of Scala Native.

  • The presenting problem was that some tests used to test Zio compiled
    with Scala native. reported failures not seen in JVM or Scala.js
    variants. Examination showed that the reported failures were because
    two Scala BigInts compared false using '==' while comparing true
    using .equals(). They should have compared true in the former case
    and did not.

    Later studies for issue __scala_.== is borked with BigInt & BigInteger, almost certainly both BigDecimal #1770 showed that comparison of two
    Scala BigDecimal instances which should have compared '=='
    as true compared as false.

    The edit to javalib/...java/lang/Number.scala addresses this problem.
    ScalaNumber extends Number.

  • I scanned the entire code base for similar problems. The family
    of Atomic*Number objects were examined more closely and found not
    to bear this flaw.

  • Unit tests were created to exercise the proposed changes. They
    failed before this PR and pass as of this PR.

Documentation:

  • This PR deserves a prominent change note under a "Bugs" fixed
    section.

Testing:

Safety
  • Built ("rebuild")and tested ("test-all") in debug mode using sbt 1.3.10
    & Java 8 on X86_64 only . All tests pass.
Efficacy
  • Specially crafted unit-tests provided in this PR fail before the PR and pass after it.

Comment on lines 1287 to 1289
@inline override def __scala_==(other: _Object): scala.Boolean = {
this.equals(other)
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not be here. The codegen is supposed to automatically add this in the vtable because BigDecimal overrides equals, as explained by the comment at

@inline def __scala_==(that: _Object): scala.Boolean = {
// This implementation is only called for classes that don't override
// equals. Otherwise, whenever equals is overriden, we also update the
// vtable entry for scala_== to point to the override directly.
this eq that
}

If that's not the case, that's a problem in the codegen; not in the library.

In general, user space classes should never implement __scala_==, since that is an implementation detail of Scala Native.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

Comment on lines 465 to 468
@inline override def __scala_==(other: _Object): scala.Boolean = {
this.equals(other)
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as the comment in BigDecimal above.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

Comment on lines 14 to 19
if (other.isInstanceOf[ScalaNumber] && !this.isInstanceOf[ScalaNumber]) {
if (this.isInstanceOf[ScalaNumber]) {
this.equals(other)
} else if (other.isInstanceOf[ScalaNumber]) {
other.equals(this)
} else {
super.__scala_==(other)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is the appropriate fix. The previous code should have been correct, if java.lang.Object.__scala_== had a reasonable implementation. The reasonable implementation should be:

  @inline def __scala_==(that: _Object): scala.Boolean =
    this.equals(that)

Instead, it's

@inline def __scala_==(that: _Object): scala.Boolean = {
// This implementation is only called for classes that don't override
// equals. Otherwise, whenever equals is overriden, we also update the
// vtable entry for scala_== to point to the override directly.
this eq that
}

as an optimization based on some vtable hackery.

Unfortunately the vtable hackery doesn't patch up super.__scala_== calls. That means we can never do super.__scala_==, and instead we should inline the correct, reasonable implementation that is written nowhere.

So the correct implementation of Number.__scala_== should be:

  @inline override def __scala_==(other: _Object): scala.Boolean = {
    if (other.isInstanceOf[ScalaNumber] && !this.isInstanceOf[ScalaNumber]) {
      other.equals(this)
    } else {
      this.equals(other)
    }
  }

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I committed with the recommended fix and Travis CI is all green. Thank you for the
patient explanation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After the successful commit, I sat down and drew out the truth tables, mentally thanking my Discrete Math professor.

I believe the following code is equivalent and correct. It passes the same tests as the
code in the current commit. I find this code to be easier to read and understand.
I think future developers may also find it easier.

I can do another commit if my proposed code is acceptable.

  @inline override def __scala_==(other: _Object): scala.Boolean = {           
    if ((this.isInstanceOf[ScalaNumber]) ||                                    
        (!other.isInstanceOf[ScalaNumber])) {                                  
      this.equals(other)                                                       
    } else {                                                                   
      other.equals(this)                                                       
    }       

It may also have performance benefits. I believe that the this left comparand will more
likely be a ScalaNumber, by a large factor. The code above would do one test in that case,
leading to a lower amortized cost for some/most workloads.

Since the code is inlined, and the type of this is known at compile time, there is
a sporting chance that the inliner will reduce the call to this.equals(other) when
this is known to be a ScalaNumber.

Normally, I would go with the commited code and be done with it. However, this is a
hot path, where likely instruction path length matters, as long as correctness is maintained.

Insert the classical refrain of "Show me the data, across a series of workloads'.

Note to future self:

The test for other being a scala number is necessary to pass the 'j.l.Integer == BigInt case and similar in the test suite. The code above can not be correctly reduced tothis.equals(other)`.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the following code is equivalent and correct. It passes the same tests as the
code in the current commit. I find this code to be easier to read and understand.
I think future developers may also find it easier.

I can do another commit if my proposed code is acceptable.

  @inline override def __scala_==(other: _Object): scala.Boolean = {           
    if ((this.isInstanceOf[ScalaNumber]) ||                                    
        (!other.isInstanceOf[ScalaNumber])) {                                  
      this.equals(other)                                                       
    } else {                                                                   
      other.equals(this)                                                       
    }       

This is indeed correct. I'm not sure about the performance impact.

It may also have performance benefits. I believe that the this left comparand will more
likely be a ScalaNumber, by a large factor. The code above would do one test in that case,
leading to a lower amortized cost for some/most workloads.

The cases where this is a ScalaNumber are basically only for scala.math.BigInt and scala.math.BigDecimal. For those, we know that equality is expensive (at least it's more than one or two tests) so their equality will dwarf whatever optimization we try to do for them in __scala_==. Therefore, this does not seem to be a scenario worth optimizing for.

Since the code is inlined, and the type of this is known at compile time, there is
a sporting chance that the inliner will reduce the call to this.equals(other) when
this is known to be a ScalaNumber.

That is true, but the same reasoning applies to the existing code as well. If this is known to be a ScalaNumber at compile time, then the condition

other.isInstanceOf[ScalaNumber] && !this.isInstanceOf[ScalaNumber]

reduces to

other.isInstanceOf[ScalaNumber] && false

which, because isInstanceOf is always pure, can also be optimized down to

false

allowing the whole method to reduce to this.equals(that).

@sjrd
Copy link
Collaborator

sjrd commented May 19, 2020

Also, could you rebase on the latest master? It seems some changes are scalafmt applied to otherwise untouched files, which shouldn't happen if the PR is up-to-date with master.

…numbers

  * This PR was motivated by Issue scala-native#1770:
    "__scala_== is borked with BigInt & BigInteger, both BigDecimal".

    That issue is now resolved.

 *  This PR needs review from experts in Scala & Java equality. It
    affects a key and widely used piece of Scala Native.

 *  The presenting problem was that some tests used to test Zio compiled
    with Scala native. reported failures not seen in JVM or Scala.js
    variants.  Examination showed that the reported failures were because
    two Scala BigInts compared false using '==' while comparing true
    using .equals().  They should have compared true in the former case
    and did not.

    Later studies for issue scala-native#1770 showed that comparison of two
    Scala BigDecimal instances which should have compared '=='
    as true compared as false.

    The edit to javalib/...java/lang/Number.scala addresses this branch
    of the problem. ScalaNumber extends Number. The proposed
    code says "If either comparand is a ScalaNumber, use the Scala
    practice of calling .equals". This gives content equality.
    "If neither comparand is a ScalaNumber, use the Java practice
    of comparing for reference equality".

    The first clause of the proposed should be non-controversial:
    if the left operand is a ScalaNumber, use Scala rules. Biasing
    towards the left seems to be a consistent and widely accepted
    approach in the readings I have done, but I do not have a Ph.D.
    in computer languages.

    The second clause derived from the long standing Scala Native code
    prior to this PR. It says: If the right operand is a ScalaNumber,
    use Scala content equality.   When I removed that clause, I could
    not replicate the comparison behaviors I saw on Scala JVM (Scalastie).

    The third clause says: If neither is a ScalaNumber, let the parent
    handle it. Object, the parent, implements __scala_== as reference
    equality (eq), the Java norm.

  * The edits to BigInteger.scala & BigDecimal.scala fix an independent
    but conceptually related problem.

    In Scala, two Java BigIntegers compared using '==' should be compared
    using content equality: equals(). Similarly, two Java BigDecimals
    should be compared using content equality.

  * I scanned the entire code base for similar problems.  The family
    of Atomic*Number objects were examined more closely and found not
    to bear this flaw.

  * Unit tests were created to exercise the proposed changes.  They
    failed before this PR and pass as of this PR.

    Whilst developing the PR, I checked comparing elements of the
    set (BigInt, ScalaBigDecimal, BigInteger, JavaBigDecimal) to other
    numeric types and primitives not of their kind.  I did not include
    tests for those, as the comparisons all turn out to be always false.

X#### Documentation:

  * This PR deserves a prominent change note under a "Bugs" fixed
    section.

X#### Testing:

X###### Safety

  * Built ("rebuild")and tested ("test-all") in debug mode using sbt 1.3.10
    & Java 8 on X86_64 only . All tests pass.

X###### Efficacy

  * Specially crafted unit-tests fail before the PR and pass after it.
@sjrd sjrd changed the title Fix #1770: __scala_== comparisons using Scala & Java Big numbers Fix #1770: Fix __scala_== with ScalaNumbers. May 25, 2020
@sjrd sjrd merged commit 83c86ef into scala-native:master May 25, 2020
@eed3si9n eed3si9n mentioned this pull request Nov 29, 2020
ekrich pushed a commit to ekrich/scala-native that referenced this pull request May 21, 2021
…e#1805)

The implementation of `java.lang.Number.__scala_==` was correct in
spirit, but failed in practice because of quirks of the linker and
`java.lang.Object.__scala_==`.

Normally, `Object.__scala_==` should be implemented as
`this.equals(that)`. For performance reasons, it is instead
implemented as `this eq that`, with the understanding that the
linker will patch the vtable of any class that overrides `equals`
so that `__scala_==` refers to that `equals`.

The issue is that this patching does not apply to super calls of the
form `super.__scala_==`, making any such call incorrect.

We fix the issue by directly calling `this.equals(that)` instead of
`super.__scala_==`, since the former would be the reasonable
implementation of `Object.__scala_==`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants