New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add implicit function types #1775

Merged
merged 37 commits into from Dec 18, 2016

Conversation

@odersky
Contributor

odersky commented Dec 5, 2016

This is the first step to bring contextual abstraction to Scala.

@edmundnoble

Excellently done. We'll beat Idris in no time. I have a few more questions:

  1. How does this interact with specialization of the FunctionN types for lower N?
  2. How does this interact with FunctionXXL?
- managing capabilities for security critical tasks,
- wiring components up with dependency injection,
- defining the meanings of operations with type classes,
- more generally, passing any sort of context to a computation.

This comment has been minimized.

@edmundnoble

edmundnoble Dec 5, 2016

Contributor

I can only see a very weak relationship between comonads and implicit functions. At best, implicit functions can replace coreader comonad stacks. Other than that, comonads are completely irrelevant.

@edmundnoble

edmundnoble Dec 5, 2016

Contributor

I can only see a very weak relationship between comonads and implicit functions. At best, implicit functions can replace coreader comonad stacks. Other than that, comonads are completely irrelevant.

This comment has been minimized.

@odersky

odersky Dec 6, 2016

Contributor

I meant to refer to the original paper "Implicit parameters: dynamic scoping with static types" (POPL2000) which speculated that the best fitting foundation of implicit parameters are comonads. But thinking about it I run into troubles to nail it down.

@odersky

odersky Dec 6, 2016

Contributor

I meant to refer to the original paper "Implicit parameters: dynamic scoping with static types" (POPL2000) which speculated that the best fitting foundation of implicit parameters are comonads. But thinking about it I run into troubles to nail it down.

This comment has been minimized.

@odersky

odersky Dec 6, 2016

Contributor

I think I have worked it out now: implicit functions are both monads and comonads, with

M[A]        =  implicit X => A
unit a      =  implicit (_: X) => a
counit m    =  m (implicitly[X])
map m f     =  implicit (_: X) => f (m (implicitly[X]))
            =  unit (f (counit m))

join m      =  counit m 
duplicate m =  unit m

WDYT?

@odersky

odersky Dec 6, 2016

Contributor

I think I have worked it out now: implicit functions are both monads and comonads, with

M[A]        =  implicit X => A
unit a      =  implicit (_: X) => a
counit m    =  m (implicitly[X])
map m f     =  implicit (_: X) => f (m (implicitly[X]))
            =  unit (f (counit m))

join m      =  counit m 
duplicate m =  unit m

WDYT?

This comment has been minimized.

@edmundnoble

edmundnoble Dec 6, 2016

Contributor

You aren't just using the implicit function there, you're also using the implicitly[X] value in counit. The comonad you present is the Store comonad (a, a -> b), and the monad is the reader monad. I believe coeffects, which are a calculus of dependencies (http://tomasp.net/academic/papers/structural/), might provide a better theoretical underpinning.

@edmundnoble

edmundnoble Dec 6, 2016

Contributor

You aren't just using the implicit function there, you're also using the implicitly[X] value in counit. The comonad you present is the Store comonad (a, a -> b), and the monad is the reader monad. I believe coeffects, which are a calculus of dependencies (http://tomasp.net/academic/papers/structural/), might provide a better theoretical underpinning.

This comment has been minimized.

@odersky

odersky Dec 6, 2016

Contributor

The monad is indeed close to the reader monad, but I fail to see that it's the Store comonad. Where is the tupling?

@odersky

odersky Dec 6, 2016

Contributor

The monad is indeed close to the reader monad, but I fail to see that it's the Store comonad. Where is the tupling?

This comment has been minimized.

@odersky

odersky Dec 7, 2016

Contributor

There's something else at play here, and I am trying to wrap my head around it. If we drop the implicitness and just talk about function abstraction and application we also get a bimonad in some sense:

M[A]        =  X => A
unit a      =  (x: X) => a
counit m    =  m x
map m f     =  (x: X) => f (m x)
            =  unit (f (counit m))

join m      =  counit m 
duplicate m =  unit m

But this feels a little bit weirder because now we are dealing with open terms, with counit operations introducing free variables that are captured by unit operations. And the names of both free variables and lambda binders are fixed to be always the same name x.

By contrast, all terms in the implicit function bimonad representation are closed. I am not sure what difference it makes, though.

@odersky

odersky Dec 7, 2016

Contributor

There's something else at play here, and I am trying to wrap my head around it. If we drop the implicitness and just talk about function abstraction and application we also get a bimonad in some sense:

M[A]        =  X => A
unit a      =  (x: X) => a
counit m    =  m x
map m f     =  (x: X) => f (m x)
            =  unit (f (counit m))

join m      =  counit m 
duplicate m =  unit m

But this feels a little bit weirder because now we are dealing with open terms, with counit operations introducing free variables that are captured by unit operations. And the names of both free variables and lambda binders are fixed to be always the same name x.

By contrast, all terms in the implicit function bimonad representation are closed. I am not sure what difference it makes, though.

This comment has been minimized.

@odersky

odersky Dec 7, 2016

Contributor

Regarding composition, I believe one can explain the composability of implicit functions from the principle that they are a bimonad. Let's say you have two function types F and G, which take implicit parameters of types X and Y, but in different orders:

type F =  implicit X => implicit Y => A
type G =  implicit Y => implicit X => A

I can turn a value f: F into a value of type G like this:

implicit Y => implicit X => f(implicitly[X])(implicitly[Y])

Moreover that conversion is completely automatic. I just have to write

val x: G  = f

and the rest is produced automatically. I believe that gets us to the essence why implicit functions are better composable than general monads. Even if we disregard the aspect of implicit code insertion, what happens here is that we strip all (co)monads in the applications to the two implicitly arguments and then reapply the (co)monads in the outer lambda abstractions. The fact that this is a bimonad is essential here, because it means we can always get out of an entangled structure with counit operations, reorder at the outermost level, and get back with unit operations.

@odersky

odersky Dec 7, 2016

Contributor

Regarding composition, I believe one can explain the composability of implicit functions from the principle that they are a bimonad. Let's say you have two function types F and G, which take implicit parameters of types X and Y, but in different orders:

type F =  implicit X => implicit Y => A
type G =  implicit Y => implicit X => A

I can turn a value f: F into a value of type G like this:

implicit Y => implicit X => f(implicitly[X])(implicitly[Y])

Moreover that conversion is completely automatic. I just have to write

val x: G  = f

and the rest is produced automatically. I believe that gets us to the essence why implicit functions are better composable than general monads. Even if we disregard the aspect of implicit code insertion, what happens here is that we strip all (co)monads in the applications to the two implicitly arguments and then reapply the (co)monads in the outer lambda abstractions. The fact that this is a bimonad is essential here, because it means we can always get out of an entangled structure with counit operations, reorder at the outermost level, and get back with unit operations.

This comment has been minimized.

@edmundnoble

edmundnoble Dec 7, 2016

Contributor

I think that your insight that the regular function bimonad requires the ability to introduce terms into the environment shows that the implicit function comonad also requires that ability. But yes, I agree that being able to implicitly reorder implicit arguments decreases their syntactic overhead below reader.

@edmundnoble

edmundnoble Dec 7, 2016

Contributor

I think that your insight that the regular function bimonad requires the ability to introduce terms into the environment shows that the implicit function comonad also requires that ability. But yes, I agree that being able to implicitly reorder implicit arguments decreases their syntactic overhead below reader.

This comment has been minimized.

@Blaisorblade

Blaisorblade Dec 7, 2016

Contributor

I agree with @edmundnoble: in particular what you get with implicits isn't closed terms, because they assume something in the context. I haven't read the paper, but maybe they had some other category in mind? (Warning: dense and unchecked ideas ahead). Beyond the category of functions you're using, you could probably have (equivalent I think) categories of expressions with x in scope and of expressions with some implicit in scope, and there your comonad might work. I haven't checked any of the details though. Whether that's insightful or useful, beyond making precise the two things are equivalent, is not clear.

@Blaisorblade

Blaisorblade Dec 7, 2016

Contributor

I agree with @edmundnoble: in particular what you get with implicits isn't closed terms, because they assume something in the context. I haven't read the paper, but maybe they had some other category in mind? (Warning: dense and unchecked ideas ahead). Beyond the category of functions you're using, you could probably have (equivalent I think) categories of expressions with x in scope and of expressions with some implicit in scope, and there your comonad might work. I haven't checked any of the details though. Whether that's insightful or useful, beyond making precise the two things are equivalent, is not clear.

This comment has been minimized.

@tpetricek

tpetricek Dec 8, 2016

I see a mention of my work on coeffects in the thread already, so I thought I'll add my perspective. I'm not all that familiar with Scala, so I'll stick to my Haskell/ML-style notation, but I hope that will be understandable.

What are coeffects
First of all, coeffects are, indeed, closely related to comonads (they are modelled using "indexed comonads", which is a generalisation of ordinary comonads). What coeffects add, is that they track more precisely what context is needed. The comonadic (or monadic) type C a or M a tells you that there is some context-requirement or some effect, but it does not tell you what precisely. Coeffects add an annotation so that you have types such as C {?p:int, ?q:int} a where the annotation {?p:int, ?q:int} says you need two implicit parameters ?p and ?q. So you can see coeffects as more precise comonads.

Are implicit parameters monads, comonads or coeffects
Implicit parameters are one of the motivating examples in the work on coeffects, but they are just one (coeffects capture other interesting contextual properties). The interesting thing about implicit parameters is that they can almost be modelled by both Reader monad and Product comonad. With monads, you use functions a -> M b for some monad and with comonads, you use functions C a -> b for some comonad.

With Reader monad, we have M a = State -> a and so:

a -> M b = a -> (State -> b)

With Product comonad, we have C a = a * State and so:

C a -> b = a * State -> b

Now you can see that the two functions, a * State -> b and a -> State -> b are related (via currying)! 🎉

There is one thing that reader monad does not let you do. It does not let you model the facts that implicit parameters (in GHC) support both lexical and dynamic scoping. Take for example this:

let ?x = 1
let f = (fun n ->?x + ?y)

If we model this using Reader monad and ?x and ?y mean "read implicit parameter from the monad", then the type of f will be a function that needs ?x and ?y. However, if we do this using comonads, the context available in the function body can combine context provided by the caller with the context provided by the declaration site - and so we can use ?x from the declaration site and end up with a function that needs just ?y (I'd be quite curious to see what Scala is doing here - are implicits just dynamically scoped or mix of both?

Links with more information
There is a lot more about this than I can fit in a comment, so for those interested:

@tpetricek

tpetricek Dec 8, 2016

I see a mention of my work on coeffects in the thread already, so I thought I'll add my perspective. I'm not all that familiar with Scala, so I'll stick to my Haskell/ML-style notation, but I hope that will be understandable.

What are coeffects
First of all, coeffects are, indeed, closely related to comonads (they are modelled using "indexed comonads", which is a generalisation of ordinary comonads). What coeffects add, is that they track more precisely what context is needed. The comonadic (or monadic) type C a or M a tells you that there is some context-requirement or some effect, but it does not tell you what precisely. Coeffects add an annotation so that you have types such as C {?p:int, ?q:int} a where the annotation {?p:int, ?q:int} says you need two implicit parameters ?p and ?q. So you can see coeffects as more precise comonads.

Are implicit parameters monads, comonads or coeffects
Implicit parameters are one of the motivating examples in the work on coeffects, but they are just one (coeffects capture other interesting contextual properties). The interesting thing about implicit parameters is that they can almost be modelled by both Reader monad and Product comonad. With monads, you use functions a -> M b for some monad and with comonads, you use functions C a -> b for some comonad.

With Reader monad, we have M a = State -> a and so:

a -> M b = a -> (State -> b)

With Product comonad, we have C a = a * State and so:

C a -> b = a * State -> b

Now you can see that the two functions, a * State -> b and a -> State -> b are related (via currying)! 🎉

There is one thing that reader monad does not let you do. It does not let you model the facts that implicit parameters (in GHC) support both lexical and dynamic scoping. Take for example this:

let ?x = 1
let f = (fun n ->?x + ?y)

If we model this using Reader monad and ?x and ?y mean "read implicit parameter from the monad", then the type of f will be a function that needs ?x and ?y. However, if we do this using comonads, the context available in the function body can combine context provided by the caller with the context provided by the declaration site - and so we can use ?x from the declaration site and end up with a function that needs just ?y (I'd be quite curious to see what Scala is doing here - are implicits just dynamically scoped or mix of both?

Links with more information
There is a lot more about this than I can fit in a comment, so for those interested:

Show outdated Hide outdated docs/blog/_posts/2016-12-05-implicit-function-types.md
Show outdated Hide outdated docs/blog/_posts/2016-12-05-implicit-function-types.md
Show outdated Hide outdated docs/blog/_posts/2016-12-05-implicit-function-types.md
Show outdated Hide outdated docs/blog/_posts/2016-12-05-implicit-function-types.md
pattern because it is lightweight and can express context changes in a
purely functional way.
The main downside of implicit parameters is the verbosity of their

This comment has been minimized.

@edmundnoble

edmundnoble Dec 5, 2016

Contributor

Just editorializing: in my opinion, the main downside of implicit parameters is that they make code harder to refactor by making the place where your code exists change the semantics of the code. The verbosity is definitely an issue on its own, though.

@edmundnoble

edmundnoble Dec 5, 2016

Contributor

Just editorializing: in my opinion, the main downside of implicit parameters is that they make code harder to refactor by making the place where your code exists change the semantics of the code. The verbosity is definitely an issue on its own, though.

This comment has been minimized.

@odersky

odersky Dec 6, 2016

Contributor

But that's nothing new. All free bindings of a piece of code affect its semantics. So the semantics of any expression that's not closed depends on its location (?)

@odersky

odersky Dec 6, 2016

Contributor

But that's nothing new. All free bindings of a piece of code affect its semantics. So the semantics of any expression that's not closed depends on its location (?)

This comment has been minimized.

@edmundnoble

edmundnoble Dec 6, 2016

Contributor

Fair point, free bindings impede refactoring whether they're implicit or not. My mistake.

@edmundnoble

edmundnoble Dec 6, 2016

Contributor

Fair point, free bindings impede refactoring whether they're implicit or not. My mistake.

This comment has been minimized.

@Blaisorblade

Blaisorblade Dec 8, 2016

Contributor

But that's nothing new. All free bindings of a piece of code affect its semantics. So the semantics of any expression that's not closed depends on its location (?)

Implicits simply hide the free variables in use, so that figuring out the context requires typechecking rather than just parsing. So, after you look up implicitly's type, you discover that implicitly[Foo] is an open term (which is sugar for open term implicitly[Foo](theImplicitDefinition)). This is confusing enough it was (indirectly) discussed above: #1775 (comment)

@Blaisorblade

Blaisorblade Dec 8, 2016

Contributor

But that's nothing new. All free bindings of a piece of code affect its semantics. So the semantics of any expression that's not closed depends on its location (?)

Implicits simply hide the free variables in use, so that figuring out the context requires typechecking rather than just parsing. So, after you look up implicitly's type, you discover that implicitly[Foo] is an open term (which is sugar for open term implicitly[Foo](theImplicitDefinition)). This is confusing enough it was (indirectly) discussed above: #1775 (comment)

@odersky

This comment has been minimized.

Show comment
Hide comment
@odersky

odersky Dec 7, 2016

Contributor

In light of the (very interesting!) discussion about monads and comonads here I think it's better to talk about "contextual" abstraction, so I have changed that line in the PR explanation. It seems like comonadic comes into play but what implicit functions provide cannot be explained exclusively (or even primarily) by their comonadicness.

Contributor

odersky commented Dec 7, 2016

In light of the (very interesting!) discussion about monads and comonads here I think it's better to talk about "contextual" abstraction, so I have changed that line in the PR explanation. It seems like comonadic comes into play but what implicit functions provide cannot be explained exclusively (or even primarily) by their comonadicness.

@liufengyun

This comment has been minimized.

Show comment
Hide comment
@liufengyun

liufengyun Dec 7, 2016

Contributor

I drafted the immature idea of dynamic parameters here, which is marginally related to this discussion.

Contributor

liufengyun commented Dec 7, 2016

I drafted the immature idea of dynamic parameters here, which is marginally related to this discussion.

@odersky odersky closed this Dec 7, 2016

@odersky odersky deleted the dotty-staging:add-implicit-funtypes branch Dec 7, 2016

@odersky odersky reopened this Dec 7, 2016

@notxcain

This comment has been minimized.

Show comment
Hide comment
@notxcain

notxcain Dec 7, 2016

Could you please provide with more examples where it is really useful (the one from article is very synthetic IMHO)? Not sure how it is different from using ReaderT for passing context. While this kind of implicits (contextual) make things harder to reason about.

notxcain commented Dec 7, 2016

Could you please provide with more examples where it is really useful (the one from article is very synthetic IMHO)? Not sure how it is different from using ReaderT for passing context. While this kind of implicits (contextual) make things harder to reason about.

@nafg

This comment has been minimized.

Show comment
Hide comment
@nafg

nafg Dec 8, 2016

Thought while reading the blog post --- what if implicitly were redefined as def implicitly[A]: implicit A => A, then instead of def thisTransaction: Transactional[Transaction] = implicitly[Transaction] you could just use type inference with def thisTransaction = implicitly[Transaction]

(OT -- is there any chance of getting polymorphic function values one day?)

nafg commented Dec 8, 2016

Thought while reading the blog post --- what if implicitly were redefined as def implicitly[A]: implicit A => A, then instead of def thisTransaction: Transactional[Transaction] = implicitly[Transaction] you could just use type inference with def thisTransaction = implicitly[Transaction]

(OT -- is there any chance of getting polymorphic function values one day?)

@acjay

This comment has been minimized.

Show comment
Hide comment
@acjay

acjay Dec 8, 2016

This seems potentially really handy, but the one thing that's bugging me a bit is that if this is the most common application, it seems like it requires some nontrivial setup. First of all, thisTransaction (and its analogs for other contexts) would have to be imported all over the place, to take advantage of that syntax. Secondly, every apparent method output type has got to be decorated with the context the method requires for input. I can see this being really confusing, especially for output types that might already involve parameterized types. It seems like it could be better to annotate the input instead, somehow. Also, instead of having to use implicitly, directly or indirectly, a syntax that has that effect might help.

Suppose I could annotate an implicit function in the type parameter list, and then access the context directly:

  def f1[*Transaction](x: Int): Int = {
    [Transaction].println(s"first step: $x")
    f2(x + 1)
  }

I'm sure this syntax that drops implicitly probably conflicts with some existing Scala syntax, but it's not hitting me right away. It would be a bold move, but on the other hand, if implicit functions are a winning play for passing context, they could be ubiquitous in real Dotty code.

@liufengyun's dynamic parameters also seem to address my concerns, when it comes to the application of implicit functions for context-passing But do we know of any other killer applications of this feature? Shooting in the dark, but perhaps the ability to eta-convert methods with implicit parameter lists, preserving the implicitness?

If I understand @nafg's request for polymorphic function values, combined with implicit function types, functions and methods would almost be fully interchangeable. Default parameters and parameter names are the only things that come to mind as being left out. But now I think I've gone fully off on a tangent.

P.S. @nafg Heh, I had a very similar thought, I think: https://twitter.com/AlanJay1/status/806685555538980864, although I didn't make the connection between it and simplifying thisTransaction.

acjay commented Dec 8, 2016

This seems potentially really handy, but the one thing that's bugging me a bit is that if this is the most common application, it seems like it requires some nontrivial setup. First of all, thisTransaction (and its analogs for other contexts) would have to be imported all over the place, to take advantage of that syntax. Secondly, every apparent method output type has got to be decorated with the context the method requires for input. I can see this being really confusing, especially for output types that might already involve parameterized types. It seems like it could be better to annotate the input instead, somehow. Also, instead of having to use implicitly, directly or indirectly, a syntax that has that effect might help.

Suppose I could annotate an implicit function in the type parameter list, and then access the context directly:

  def f1[*Transaction](x: Int): Int = {
    [Transaction].println(s"first step: $x")
    f2(x + 1)
  }

I'm sure this syntax that drops implicitly probably conflicts with some existing Scala syntax, but it's not hitting me right away. It would be a bold move, but on the other hand, if implicit functions are a winning play for passing context, they could be ubiquitous in real Dotty code.

@liufengyun's dynamic parameters also seem to address my concerns, when it comes to the application of implicit functions for context-passing But do we know of any other killer applications of this feature? Shooting in the dark, but perhaps the ability to eta-convert methods with implicit parameter lists, preserving the implicitness?

If I understand @nafg's request for polymorphic function values, combined with implicit function types, functions and methods would almost be fully interchangeable. Default parameters and parameter names are the only things that come to mind as being left out. But now I think I've gone fully off on a tangent.

P.S. @nafg Heh, I had a very similar thought, I think: https://twitter.com/AlanJay1/status/806685555538980864, although I didn't make the connection between it and simplifying thisTransaction.

@nafg

This comment has been minimized.

Show comment
Hide comment
@nafg

nafg Dec 8, 2016

nafg commented Dec 8, 2016

@liufengyun

This comment has been minimized.

Show comment
Hide comment
@liufengyun

liufengyun Dec 8, 2016

Contributor

@nafg To summarize my view: I think dynamic parameters can do better than implicit function types.

Dynamic parameters is based on the following paper (renamed to avoid confusion with Scala implicits):

Implicit parameters: dynamic scoping with static types, Jeffrey R. Lewis, POPL '00

The detailed argument is in the Gist.

Contributor

liufengyun commented Dec 8, 2016

@nafg To summarize my view: I think dynamic parameters can do better than implicit function types.

Dynamic parameters is based on the following paper (renamed to avoid confusion with Scala implicits):

Implicit parameters: dynamic scoping with static types, Jeffrey R. Lewis, POPL '00

The detailed argument is in the Gist.

@nafg

This comment has been minimized.

Show comment
Hide comment
@nafg

nafg Dec 9, 2016

Here's another possible use case.
Some libraries, like Slick and Shapeless, have a lot of machinery happening based on implicits. The problem is that if I'm writing some polymorphic code, the errors about missing implicits are often very cryptic and it's hard to know what implicit parameters my method needs, and besides it's very tedious to add things like slick.lifted.Shape[Level <: ShapeLevel, -Mixed_, Unpacked_, Packed_], or all the operations you may need on your HList. If instead they would return implicit functions, and I would wrap them via function composition, perhaps the required typeclasses could just be inferred.

nafg commented Dec 9, 2016

Here's another possible use case.
Some libraries, like Slick and Shapeless, have a lot of machinery happening based on implicits. The problem is that if I'm writing some polymorphic code, the errors about missing implicits are often very cryptic and it's hard to know what implicit parameters my method needs, and besides it's very tedious to add things like slick.lifted.Shape[Level <: ShapeLevel, -Mixed_, Unpacked_, Packed_], or all the operations you may need on your HList. If instead they would return implicit functions, and I would wrap them via function composition, perhaps the required typeclasses could just be inferred.

@odersky

This comment has been minimized.

Show comment
Hide comment
@odersky

odersky Dec 12, 2016

Contributor

For discussions about the concept of implicit parameters and the blog post, let's go to:

https://contributors.scala-lang.org

Let's narrow this discussion here on the specifics of the implementation.

Contributor

odersky commented Dec 12, 2016

For discussions about the concept of implicit parameters and the blog post, let's go to:

https://contributors.scala-lang.org

Let's narrow this discussion here on the specifics of the implementation.

@odersky

This comment has been minimized.

Show comment
Hide comment
@odersky

odersky Dec 13, 2016

Contributor

Rebased to master

Contributor

odersky commented Dec 13, 2016

Rebased to master

@odersky

This comment has been minimized.

Show comment
Hide comment
@odersky

odersky Dec 13, 2016

Contributor

The last commits represent the second step to contectual abstraction in Scala. They make
implicit function types a close to zero-cost abstraction, by means of a phase that optimizes them
to plain methods.

Contributor

odersky commented Dec 13, 2016

The last commits represent the second step to contectual abstraction in Scala. They make
implicit function types a close to zero-cost abstraction, by means of a phase that optimizes them
to plain methods.

* is expanded to two methods:
*
* def m(xs: Ts): IF = implicit (ys: Us) => m$direct(xs)(ys)
* def m$direct(xs: Ts)(ys: Us): R = E

This comment has been minimized.

@Blaisorblade

Blaisorblade Dec 13, 2016

Contributor

Any reason why this optimization is restricted to implicit functions? Done this way, this seems rather ad-hoc.

@Blaisorblade

Blaisorblade Dec 13, 2016

Contributor

Any reason why this optimization is restricted to implicit functions? Done this way, this seems rather ad-hoc.

This comment has been minimized.

@odersky

odersky Dec 13, 2016

Contributor

The main reason is that the rewrite is based on the type of the result. For implicit function types we have a perfect match. The typing rules guarantee that every function that has an implicit function result type must be implemented by a closure. For normal functions this is not guaranteed, so we do not know whether the optimization is globally effective or not. It could make things worse, actually.

@odersky

odersky Dec 13, 2016

Contributor

The main reason is that the rewrite is based on the type of the result. For implicit function types we have a perfect match. The typing rules guarantee that every function that has an implicit function result type must be implemented by a closure. For normal functions this is not guaranteed, so we do not know whether the optimization is globally effective or not. It could make things worse, actually.

This comment has been minimized.

@DarkDimius

DarkDimius Dec 13, 2016

Member

@Blaisorblade, additionally note that normal function types are non-final and in case m is overriden in as subclass by a non closure with custom apply logic, you could have hard time figuring out what should $direct method do.

@DarkDimius

DarkDimius Dec 13, 2016

Member

@Blaisorblade, additionally note that normal function types are non-final and in case m is overriden in as subclass by a non closure with custom apply logic, you could have hard time figuring out what should $direct method do.

This comment has been minimized.

@Blaisorblade

Blaisorblade Dec 13, 2016

Contributor

The typing rules guarantee that every function that has an implicit function result type must be implemented by a closure.

So I can't implement an implicit function type by returning a closure? Are those non-first-class?

Quite a few things suggest the following: A => B is a value type, while implicit A => B is a non-value type (IMHO a method type). I see why you're doing that, but that doesn't sound very orthogonal to me—this is a small thing, but that's the sort of thing which gives Scala its reputation for complexity. And while this fits in the heads of compiler hackers, the heads of (advanced) users also matter.

On Discourse I started sketching, as an alternative, a (limited) form of abstraction on non-value types (in particular, method types). I expect there are questions there too, but right now it seems to me you are in fact implementing a special-case of that.
(I don't propose abstract non-value types—I'd expect them to be type aliases, according to the same rules you use here, whatever they are exactly).

@Blaisorblade

Blaisorblade Dec 13, 2016

Contributor

The typing rules guarantee that every function that has an implicit function result type must be implemented by a closure.

So I can't implement an implicit function type by returning a closure? Are those non-first-class?

Quite a few things suggest the following: A => B is a value type, while implicit A => B is a non-value type (IMHO a method type). I see why you're doing that, but that doesn't sound very orthogonal to me—this is a small thing, but that's the sort of thing which gives Scala its reputation for complexity. And while this fits in the heads of compiler hackers, the heads of (advanced) users also matter.

On Discourse I started sketching, as an alternative, a (limited) form of abstraction on non-value types (in particular, method types). I expect there are questions there too, but right now it seems to me you are in fact implementing a special-case of that.
(I don't propose abstract non-value types—I'd expect them to be type aliases, according to the same rules you use here, whatever they are exactly).

This comment has been minimized.

This comment has been minimized.

@odersky

odersky Dec 14, 2016

Contributor

Let's say you have

 val cl: implicit A => B
 def f: implicit A => B = cl

Then by the rules of implicit expansion this will gve you an eta-expansion on the rhs of f:

def f: implicit A => B = implicit $x: A => cl($x)

That's what guarantees that the right-hand side of every method with implicit function result type is an implicit function value. For normal functions that guarantee does not hold, so we might well be pessimizing code with a shortcutting optimization.

@odersky

odersky Dec 14, 2016

Contributor

Let's say you have

 val cl: implicit A => B
 def f: implicit A => B = cl

Then by the rules of implicit expansion this will gve you an eta-expansion on the rhs of f:

def f: implicit A => B = implicit $x: A => cl($x)

That's what guarantees that the right-hand side of every method with implicit function result type is an implicit function value. For normal functions that guarantee does not hold, so we might well be pessimizing code with a shortcutting optimization.

This comment has been minimized.

@Blaisorblade

Blaisorblade Dec 15, 2016

Contributor

Ah, so here we inline the eta-expansion and emit a call to cl, while with normal functions we only have a non-inlined call to CL (which will be inlined anyway by the JVM). Fair enough.
Maybe some info about this should be part of the code docs?

@Blaisorblade

Blaisorblade Dec 15, 2016

Contributor

Ah, so here we inline the eta-expansion and emit a call to cl, while with normal functions we only have a non-inlined call to CL (which will be inlined anyway by the JVM). Fair enough.
Maybe some info about this should be part of the code docs?

@odersky

This comment has been minimized.

Show comment
Hide comment
@odersky

odersky Dec 14, 2016

Contributor

/rebuild

Contributor

odersky commented Dec 14, 2016

/rebuild

case Block(stats, expr) => cpy.Block(tree)(stats, directQual(expr))
case tree: RefTree =>
cpy.Ref(tree)(tree.name.directName)
.withType(directMethod(tree.symbol).termRef)

This comment has been minimized.

@retronym

retronym Dec 15, 2016

Contributor

I'm guessing that this transform is (correctly) skipped for:

scala> def foo(a: Any): Transactional[Int] = 42
def foo(a: Any): Transactional[Int]
scala> (if ("".isEmpty) foo("") else foo("")).apply("")

Because tree.qualifier.symbol would be NoSymbol.

But just wanted to check that I'm reading this correctly.

@retronym

retronym Dec 15, 2016

Contributor

I'm guessing that this transform is (correctly) skipped for:

scala> def foo(a: Any): Transactional[Int] = 42
def foo(a: Any): Transactional[Int]
scala> (if ("".isEmpty) foo("") else foo("")).apply("")

Because tree.qualifier.symbol would be NoSymbol.

But just wanted to check that I'm reading this correctly.

This comment has been minimized.

@odersky

odersky Dec 15, 2016

Contributor

Yes, that's correct.

@odersky

odersky Dec 15, 2016

Contributor

Yes, that's correct.

flags = sym.flags | Synthetic,
info = directInfo(sym.info))
if (direct.allOverriddenSymbols.isEmpty) direct.resetFlag(Override)
direct

This comment has been minimized.

@retronym

retronym Dec 16, 2016

Contributor

Does this force the current info transform on the base types? If not, the result might be non-determistic. I found this a bit fiddly to get right in an example i once created for how to do this in a compiler plugin.

@retronym

retronym Dec 16, 2016

Contributor

Does this force the current info transform on the base types? If not, the result might be non-determistic. I found this a bit fiddly to get right in an example i once created for how to do this in a compiler plugin.

This comment has been minimized.

@DarkDimius

DarkDimius Dec 16, 2016

Member

It will invoke infoTransforms, but this phase doesn't register one(it actually does, but it's identity), it forcefully updates denotations.

@DarkDimius

DarkDimius Dec 16, 2016

Member

It will invoke infoTransforms, but this phase doesn't register one(it actually does, but it's identity), it forcefully updates denotations.

This comment has been minimized.

@odersky

odersky Dec 16, 2016

Contributor

@retronym Any infoTransforms in phases up to this one would be forced on the basetypes, yes.

@odersky

odersky Dec 16, 2016

Contributor

@retronym Any infoTransforms in phases up to this one would be forced on the basetypes, yes.

}
val (remappedCore, fwdClosure) = splitClosure(mdef.rhs)
val originalDef = cpy.DefDef(mdef)(rhs = fwdClosure)

This comment has been minimized.

@retronym

retronym Dec 16, 2016

Contributor

Do you need to reset the ABSTRACT flag on the original method, now that it always contains a forwarder?

@retronym

retronym Dec 16, 2016

Contributor

Do you need to reset the ABSTRACT flag on the original method, now that it always contains a forwarder?

This comment has been minimized.

@odersky

odersky Dec 16, 2016

Contributor

No, a Deferred method would not get a forwarder.

@odersky

odersky Dec 16, 2016

Contributor

No, a Deferred method would not get a forwarder.

@odersky

This comment has been minimized.

Show comment
Hide comment
@odersky

odersky Dec 16, 2016

Contributor

Rebased to master

Contributor

odersky commented Dec 16, 2016

Rebased to master

@odersky

This comment has been minimized.

Show comment
Hide comment
@odersky

odersky Dec 17, 2016

Contributor

Rebased again to master. I am going to merge as soon as tests pass because I am growing tired of this.

Contributor

odersky commented Dec 17, 2016

Rebased again to master. I am going to merge as soon as tests pass because I am growing tired of this.

Add ImplicitFunctionN classes
These are always synthetic; generated on demand.

odersky added some commits Dec 5, 2016

Fixes to tests
1. I noted java_all was not running(it took 0.01s to complete); fixed by
   changing the test directory.

2. We suspected tasty_bootstrap was gettng the wrong classpath and
   had a lot of problems getting it to print the classpatg. Fixed
   by refactoring the options we pass to tasty_bootstrap (it has
   to be -verbose in addition to -classpath). For the moment,
   both a turned off but we have to just swap a false to a true
   to turn them on together.
Fix "wrong number of args" reporting
"Wrong number of args" only works for type arguments but was called also for
term arguments. Ideally we should have a WrongNumberOfArgs message that works for
both, but this will take some refactoring.
Ref copier that works for Idents and Selects
The Ref copier copies Idents and Selects, changing the name
of either.
initialDenot method for symbols
This avoids denotation transforms when called at a later
phase because it cuts out current. Not needed in final
version of ShortcutImplicits, but I thought it was
good to have.
New ShortcutImplicits phase
Optimizes implicit closures by avoiding closure
creation where possible.
Add benchmarks
Benchmark code to compare compilation schemes in
different scenarios. See results.md for explanations.
Make specialization tweakable
Introduce an option to not specialize monomorphic
targets of callsites.
Drop Override flag for non-overriding direct methods
Also, integrate Jason's test case with the conditional.
@odersky

This comment has been minimized.

Show comment
Hide comment
@odersky

odersky Dec 17, 2016

Contributor
Contributor

odersky commented Dec 17, 2016

@odersky odersky merged commit 7866bc2 into lampepfl:master Dec 18, 2016

6 checks passed

cla @odersky signed the Scala CLA. Thanks!
Details
continuous-integration/drone/pr the build was successful
Details
validate-junit [26] SUCCESS. Took 31 min.
Details
validate-main [26] SUCCESS. Took 31 min.
Details
validate-partest [26] SUCCESS. Took 28 min.
Details
validate-partest-bootstrapped [26] SUCCESS. Took 17 min.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment