New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scala Wart: Weak eta-expansion #2570

Closed
lihaoyi opened this Issue May 29, 2017 · 20 comments

Comments

Projects
None yet
9 participants
@lihaoyi
Copy link
Contributor

lihaoyi commented May 29, 2017

Opening this issue, as suggested by Martin, to provide a place to discuss the individual warts brought up in the blog post Warts of the Scala Programming Language and the possibility of mitigating/fixing them in Dotty (and perhaps later in Scala 2.x). These are based on Scala 2.x behavior, which I understand Dotty follows closely, apologies in advance if it has already been fixed


Scala maintains a distinction between "functions" and "methods": in general,
methods are things you call on an object, whereas functions are objects
themselves. However, since they're so similar ("things you can call"), it gives
you a way to easily wrap a method in a function object called "eta expansion"

@ def repeat(s: String, i: Int) = s * i
defined function repeat

@ repeat("hello", 2)
res89: String = "hellohello"

@ val func = repeat _
func: (String, Int) => String = $sess.cmd90$$$Lambda$2796/1082786554@2a3983b9

@ func("hello", 3)
res91: String = "hellohellohello"

Above, we use the underscore _ to assign repeat _ to a value func, which
is then a function object we can call. This can happen automatically, without the
_, based on the "expected type" of the place the method is being used. For
example, if we expect func to be a (String, Int) => String, we can assign
repeat to it without the _:

@ val func: (String, Int) => String = repeat
func: (String, Int) => String = $sess.cmd92$$$Lambda$2803/669946146@46226d53

@ func("hello", 3)
res92: String = "hellohellohello"

Or by stubbing out the arguments with _ individually:

@ val func = repeat(_, _)
func: (String, Int) => String = $sess.cmd98$$$Lambda$2832/1025548997@358b1f86

This works, but has a bunch of annoying limitations. Firstly, even though you
can fully convert the method repeat into a (String, Int) => String value
using _, you cannot partially convert it:

@ val func = repeat("hello", _)
cmd4.sc:1: missing parameter type for expanded function 
((x$1: <error>) => repeat("hello", x$1))
val func = repeat("hello", _)
                           ^
Compilation Failed

Unless you know the the "expected type" of func, in which case you can
partially convert it:

@ val func: Int => String = repeat("hello", _)
func: Int => String = $sess.cmd93$$$Lambda$2808/1138545802@2c229ed2

Or you provide the type to the partially-applied-function-argument _ manually:

@ repeat("hello", _: Int)
res4: Int => String = $sess.cmd4$$$Lambda$1988/1407003104@5eadc347

This is a bit strange to me. If I can easily convert the entire repeat method
into a function without specifying any types, why can I not convert it into a
function if I already know one of the arguments? After all, I have provided
strictly more information in the repeat("hello", _) case than I have in the
repeat(_, _) case, and yet somehow type inference got worse!

Furthermore, there's a more fundamental issue: if I know that repeat is a
method that takes two arguments, why can't I just do this?

@ val func = repeat
cmd99.sc:1: missing argument list for method repeat in object cmd88
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `repeat _` or `repeat(_,_)` instead of `repeat`.
val func = repeat
           ^
Compilation Failed

After all, since the compiler already knows that repeat is a method, and that
it doesn't have it's arguments provided, why not convert it for me? Why force me
to go through the _ or (_, _) dance, or why ask me to provide an expected type
for func if it already knows the type of repeat?

In other languages with first-class functions, like Python, this works fine:

>>> def repeat(s, i):
...     return s * i
...

>>> func = repeat

>>> func("hello", 3)
'hellohellohello'

The lack of automatic eta-expansion results in people writing weird code to work
around it, such as this example from ScalaMock:

"drawLine" should "interact with Turtle" in {
  // Create mock Turtle object
  val mockedTurtle = mock[Turtle]
 
  // Set expectations
  (mockedTurtle.setPosition _).expects(10.0, 10.0)
  (mockedTurtle.forward _).expects(5.0)
  (mockedTurtle.getPosition _).expects().returning(15.0, 10.0)
 
  // Exercise System Under Test
  drawLine(mockedTurtle, (10.0, 10.0), (15.0, 10.0))
}

Here, the weird (foo _) dance is something that they have to do purely
because of this restriction in eta-expansion.

While I'm sure there are good implementation-reasons why this doesn't work, I
don't see any reason this shouldn't work from a language-semantics point of
view. From a user's point of view, methods and functions are just "things you
call", and Scala is generally successful and not asking you to think about the
distinction between them.

However, in cases like this, I think there isn't a good reason the compiler
shouldn't try a bit harder to figure out what I want before giving up and
asking me to pepper _s or expected types all over the place. The compiler
already has all the information it needs - after all, it works if you put
an _ after the method - and it just needs to use that information when
the _ isn't present.

@senia-psm

This comment has been minimized.

Copy link
Contributor

senia-psm commented May 29, 2017

Another use case: ask pattern for typed actors in akka.
One can use it this way:

val result: Future[Result] = actor ? (Message("msg", _))
complete(result)

but not this way:

complete {
  actor ? Message("msg", _)
}

See this example

@dragos

This comment has been minimized.

Copy link

dragos commented May 30, 2017

After all, since the compiler already knows that repeat is a method, and that
it doesn't have it's arguments provided, why not convert it for me? Why force me
to go through the _ or (_, _) dance, or why ask me to provide an expected type
for func if it already knows the type of repeat?

If I remember correctly it used to work like this but it was too error-prone when used on Java methods. In particular toString that people like to call without the (). So code like this did something unexpected (can you guess?):

"My foo: " + foo.toString

or even better:

if ("foo" == foo.toString) ...

In both cases eta-expansion wasn't what you wanted. Perhaps the second one can be ruled-out based on the expected type, or perhaps requiring parenthesis (which I totally agree it's a wart) would help, but people might still make the mistake. So should both these cases be an error? What error should it be?

@lihaoyi

This comment has been minimized.

Copy link
Contributor

lihaoyi commented May 30, 2017

I personally don't think your examples are that unexpected; we pass around Function0[T]s all the time, and if you forget to call them, well, it behaves differently than if you did call them. People sometimes accidentally sending Akka actors companion-objects rather than instantiated messages. I've spent hours during refactoring tracking down places where I'm comparing (_: T) == (_: Some[T]). I even have Some(Hello World Blog) unintentionally as the title of one of my Scala-generated blog posts in Disqus!

screenshot 2017-05-30 14 12 45

These are all basically the same problem: that any2StringAdd, String#+(x: Any), .toString, ==, and others are type-unsafe. That could be solved by allowing some kind of SuperAny that doesn't have any methods, and making Any a subclass of that, though that will be a pretty large upheaval.

Nevertheless, I think it's strange to have a language feature propagate through every codebase in the world just because people make mistakes when calling a single method .toString. If we really wanted to, we could overload .toString via compiler magic so it has both .toString and .toString() versions. Or, as mentioned in #2571, we could perform that compiler magic for any Java methods (and Scala methods with a magic annotation), so it'll just become a Java interop quirk rather than a Scala wart. So even if we decide that people forgetting ()s during Java interop is a problem, there are solutions that could avoid that problem while removing all the weirdness from pure-Scala code

@dragos

This comment has been minimized.

Copy link

dragos commented May 30, 2017

Nevertheless, I think it's strange to have a language feature propagate through every codebase in the world just because people make mistakes when calling a single method .toString

toString is just the simplest example, it's for sure not the only one! It's a thorny issue: silently accept erroneous code, or ask people to be explicit in some cases. Beginners will get this wrong and get burned at runtime, and the fact that Akka or other places in the language have similar gotchas still doesn't make it a clear win, I'd say.

Maybe the rule can be relaxed to require the _ only when the method takes no parameters?

@Jasper-M

This comment has been minimized.

Copy link
Contributor

Jasper-M commented May 30, 2017

Whatever you do, with automatic eta-expansion you will always have the special case of methods without parameter lists (def foo = "foo") which cannot be automatically converted. Even for def foo() = "foo" automatic eta-expansion is deprecated apparently.

@adriaanm

This comment has been minimized.

Copy link
Contributor

adriaanm commented Jun 3, 2017

Maybe the rule can be relaxed to require the _ only when the method takes no parameters?

That's the plan set in motion by scala/scala#5327

@SethTisue

This comment has been minimized.

Copy link
Member

SethTisue commented Jun 3, 2017

I'm leaning against this change.

It makes more things that more or less make sense, compile. Isn't that appealing? Sure, but it's a dangerous appeal. The more things compile, the more ways you can screw up:

  • ...when you're a beginner and don't really know what you're doing yet
  • ...when you're refactoring and you want the compiler to rule out meanings you didn't intend

as @lrytz pointed out in conversation just now, an example of the latter is a refactoring that adds arguments to a method that didn't have them before. now .foo still compiles, but is no longer a method call. if you're lucky the types won't line up and you'll get an error, if not, not.

admittedly none of this is conclusive, but I think we should be very cautious about adding yet more ways of writing the same thing to the language, and that's what this change does.

@odersky

This comment has been minimized.

Copy link
Contributor

odersky commented Jun 5, 2017

If we really wanted to, we could overload .toString via compiler magic so it has both .toString and .toString() versions.

toString is just used as an example. There are many more such methods. length is even worse than toString in that some of its usages are fields (i.e. in on arrays) and others are methods and then would require a (). I would guess that the majority of nullary methods in Java should be expressed without ().

@odersky

This comment has been minimized.

Copy link
Contributor

odersky commented Jun 5, 2017

After discussing this with @adriaanm, we are leaning towards the following compromise proposal for handling references to unapplied methods m:

  1. If m has one or more parameters, we always eta expand

  2. if m is nullary (i.e. has type ()R):

    1. If the expected type is of the form () => T, we eta expand.
    2. If m is defined by Java, or overrides a Java defined method, we insert ().
    3. Otherwise we issue an error of the form:

    Unapplied nullary methods are only converted to functions when a function type is expected.
    You need to either apply the method to (), or convert it to a function with () => m().

  3. The syntax m _ is deprecated.

@smarter

This comment has been minimized.

Copy link
Member

smarter commented Jun 5, 2017

Interesting! I love the idea of having one less meaning for _. What about implicit function types, do methods with implicit parameter lists ever get automatically converted to them?

@lihaoyi

This comment has been minimized.

Copy link
Contributor

lihaoyi commented Jun 5, 2017

we are leaning towards the following compromise proposal for handling references to unapplied methods m

👍

I'd personally prefer to eta-expand () Scala methods too, but I'd be happy with an error message.

I suppose def foo = ... functions without arguments will now need to be called with () => foo as well, with foo _ being deprecated as well?

EDIT: I guess the whole eta-expand-nullary-methods thing was so controversial everybody forgot about the earlier case: what do people think about being able to expand

def repeat(s: String, i: Int) = s * i
val partial = repeat("hello", _)

without an expected type?

@adriaanm

This comment has been minimized.

Copy link
Contributor

adriaanm commented Jun 5, 2017

I suppose def foo = ... functions without arguments will now need to be called with () => foo as well, with foo _ being deprecated as well?

Yes, a reference foo to a definition def foo, would not be eta-expanded implicitly, and your only option would be to write the expansion out yourself: () => foo. For def foo(), you could write foo: (() => T) (or have the type ascription be implied by the surrounding code).

@adriaanm

This comment has been minimized.

Copy link
Contributor

adriaanm commented Jun 5, 2017

repeat("hello", _)

Martin and I have both been working on an implementation for this :-)

@odersky

This comment has been minimized.

Copy link
Contributor

odersky commented Jun 6, 2017

Martin and I have both been working on an implementation for this :-)

See #2691.

@odersky

This comment has been minimized.

Copy link
Contributor

odersky commented Jun 6, 2017

Interesting! I love the idea of having one less meaning for _. What about implicit function types, do methods with implicit parameter lists ever get automatically converted to them?

No. methods with implicit parameter lists always get applied to implicit arguments.

@smarter

This comment has been minimized.

Copy link
Member

smarter commented Jun 6, 2017

But what about def foo(x: A)(implicit bla: B): C, does it become A => B => C or A => implicit B => C ?

@odersky

This comment has been minimized.

Copy link
Contributor

odersky commented Jun 6, 2017

It becomes A => C, same as what would happen if you write foo _ now.

odersky added a commit to dotty-staging/dotty that referenced this issue Jun 6, 2017

odersky added a commit to dotty-staging/dotty that referenced this issue Jun 6, 2017

odersky added a commit to dotty-staging/dotty that referenced this issue Jun 8, 2017

odersky added a commit to dotty-staging/dotty that referenced this issue Jun 8, 2017

Only add () to calls of nullary methods if they come from Java or Scala2
Follows the scheme outlined in lampepfl#2570. The reason not to flag bad calls to
methods compiled from Scala 2 is that some Scala 2 libraries are not yet in
a form where the parentheses are as they should be. For instance

    def isWhole()

in some of the numeric libraries should not have the ().

odersky added a commit to dotty-staging/dotty that referenced this issue Jun 9, 2017

odersky added a commit to dotty-staging/dotty that referenced this issue Jun 9, 2017

Only add () to calls of nullary methods if they come from Java or Scala2
Follows the scheme outlined in lampepfl#2570. The reason not to flag bad calls to
methods compiled from Scala 2 is that some Scala 2 libraries are not yet in
a form where the parentheses are as they should be. For instance

    def isWhole()

in some of the numeric libraries should not have the ().
@willisjtc

This comment has been minimized.

Copy link

willisjtc commented Jun 9, 2017

Why not just do as java and reference functions with the double colon operator or something similar and anytime you want the function to be executed you just give it the name. For example,
object::length returns the function and
object.length calls the property/function

odersky referenced this issue Jun 13, 2017

Merge pull request #2691 from dotty-staging/fix-#2570-1
Allow partial eta expansion

odersky added a commit to dotty-staging/dotty that referenced this issue Jun 13, 2017

odersky added a commit to dotty-staging/dotty that referenced this issue Jun 13, 2017

Only add () to calls of nullary methods if they come from Java or Scala2
Follows the scheme outlined in lampepfl#2570. The reason not to flag bad calls to
methods compiled from Scala 2 is that some Scala 2 libraries are not yet in
a form where the parentheses are as they should be. For instance

    def isWhole()

in some of the numeric libraries should not have the ().

odersky added a commit to dotty-staging/dotty that referenced this issue Jun 14, 2017

odersky referenced this issue Jun 23, 2017

odersky added a commit to dotty-staging/dotty that referenced this issue Jun 23, 2017

Only add () to calls of nullary methods if they come from Java or Scala2
Follows the scheme outlined in lampepfl#2570. The reason not to flag bad calls to
methods compiled from Scala 2 is that some Scala 2 libraries are not yet in
a form where the parentheses are as they should be. For instance

    def isWhole()

in some of the numeric libraries should not have the ().

@odersky odersky closed this in d1f7968 Jun 23, 2017

@lihaoyi

This comment has been minimized.

Copy link
Contributor

lihaoyi commented Jun 24, 2017

The linked diff appears to also enforce the correct-number-of-empty-parens; should #2571 be closed as well?

@smarter

This comment has been minimized.

Copy link
Member

smarter commented Jun 24, 2017

Yes indeed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment