New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

An Alternative to Implicits #5458

Open
wants to merge 35 commits into
base: master
from

Conversation

Projects
None yet
@odersky
Contributor

odersky commented Nov 16, 2018

[This PR was called "Explore Witnesses as an Alternative to Implicits"]. The current iteration replaces witness with instance. I have therefore changed the wording in the title and description of the PR]

Excerpted from the motivation.md file in this PR:

Scala's implicits are its most distinguished feature. They are the fundamental way to abstract over context. They represent a single concept with an extremely varied number of use cases, among them: implementing type classes, establishing context, dependency injection, expressing capabilities, computing new types and proving relationships between them.

At the same time, implicits are also a controversial feature. I believe there are several reasons for this.

First, being very powerful, implicits are easily over-used and mis-used. This observation holds in almost all cases when we talk about implicit conversions, which, even though conceptually different, share the same syntax with other implicit definitions. For instance,
regarding the two definitions

implicit def i1(implicit x: T): C[T] = ...
implicit def i2(x: T): C[T] = ...

the first of these is a conditional implicit value, the second an implicit conversion. Conditional implicit values are a cornerstone for expressing type classes, whereas most applications of implicit conversions have turned out to be of dubious value. The problem is that many newcomers to the language start with defining implicit conversions since they are easy to understand and seem powerful and convenient. Scala 3 will put under a language flag both definitions and applications of "undisciplined" implicit conversions between types defined elsewhere. This is a useful step to push back against overuse of implicit conversions. But the problem remains that syntactically, conversions and values just look too similar for comfort.

Second, implicits pose challenges for tooling. The set of available implicits depends on context, so command completion has to take context into account. This is feasible in an IDE but docs like ScalaDoc that are based static web pages can only provide an approximation. Another problem is that failed implicit searches often give very unspecific error messages, in particular if some deeply recursive implicit search has failed. The dotty compiler implements some improvements in this case, but further progress would be desirable.

Third, the syntax of implicit definitions might be a bit too minimal. It consists of a single modifier, implicit, that can be attached to a large number of language constructs. A problem with this for newcomers is that it often conveys mechanism better than intent. For instance, a typeclass instance is an implicit object or val if unconditional and an implicit def with implicit parameters if conditional. This describes precisely what the implicit definitions translate to -- just drop the implicit modifier, and that's it! But the cues that define intent are rather indirect and can be easily misread, as demonstrated by the definitions of i1 and i2 above.

Fourth, the syntax of implicit parameters has also some shortcomings. It starts with the position of implicit as a pseudo-modifier that applies to a whole parameter section instead of a single parameter. This represents an irregular case wrt to the rest of Scala's syntax. Furthermore, while implicit parameters are designated specifically, arguments are not. Passing an argument to an implicit parameter looks like a regular application f(arg). This is problematic because it means there can be confusion regarding what parameter gets instantiated in a call. For instance, in

def currentMap(implicit ctx: Context): Map[String, Int]

one cannot write currentMap("abc") since the string "abc" is taken as explicit argument to the implicit ctx parameter. One has to write currentMap.apply("abc") instead, which is awkward and irregular. For the same reason, a method definition can only have one implicit parameter section and it must always come last. This restriction not only reduces orthogonality, but also prevents some useful program constructs, such as a method with a regular parameter whose type depends on an implicit value. Finally, it's also a bit annoying that implicit parameters must have a name, even though in many cases that name is never referenced.

None of the shortcomings is fatal, after all implicits are very widely used, and many libraries and applications rely on them. But together, they make code using implicits more cumbersome and less clear than it could be.

Can implicit function types help? Implicit function types allow to abstract over implicit parameterization. They are a key part of the program to make as many aspects of methods as possible first class. Implicit function types can avoid much of the repetition in programs that use implicits widely. But they do not directly address the issues mentioned here.

Alternative Design

implicit is a modifier that gets attached to various constructs. I.e. we talk about implicit vals, defs, objects, parameters, or arguments. This conveys mechanism rather than intent. What is the intent that we want to convey? Ultimately it's "trade types for terms". The programmer specifies a type and the compiler fills in the term matching that type automatically. So the concept we are after would serve to express definitions that provide the canonical instances for certain types.

This PR works out this idea in detail. It consists of

  • 5 doc files in docs/docs/reference/instances
  • an implementation of what is described in the docs
  • updated test files

The PR is based on #5114

Direct link to docs:

  1. Motivation

  2. Instance Definitions

  3. Context Parameters and Arguments

  4. Replacing Implicits

  5. Discussion

@dotty-bot

Hello, and thank you for opening this PR! 🎉

All contributors have signed the CLA, thank you! ❤️

Commit Messages

We want to keep history, but for that to actually be useful we have
some rules on how to format our commit messages (relevant xkcd).

Please stick to these guidelines for commit messages:

  1. Separate subject from body with a blank line
  2. When fixing an issue, start your commit message with Fix #<ISSUE-NBR>:
  3. Limit the subject line to 72 characters
  4. Capitalize the subject line
  5. Do not end the subject line with a period
  6. Use the imperative mood in the subject line ("Add" instead of "Added")
  7. Wrap the body at 80 characters
  8. Use the body to explain what and why vs. how

adapted from https://chris.beams.io/posts/git-commit

Have an awesome day! ☀️

@odersky odersky force-pushed the dotty-staging:add-witness branch 2 times, most recently from 178bd77 to 25b84a9 Nov 16, 2018

@odersky odersky changed the title from Explore Witnesses to Explore Witnesses as an Alternative to Implicits Nov 16, 2018

@lihaoyi

This comment has been minimized.

Contributor

lihaoyi commented Nov 17, 2018

Reading through, at a first glance this seems like a mostly sideways change that isn't clearly better or worse than the status quo, and for something with such a huge & clear migration cost it should have correspondingly huge & clear benefits.

It's not clear to me that the proposed syntax/semantics significantly helps solve the problems stated up front:

  1. Is defining too many implicit conversions really a problem right now?

    • As a library user I'd argue that the most problematic usage patterns are people defining too many extension methods, which this proposal seems to encourage.
    • While this proposal makes defining extension methods somewhat easier, AFAICT it does not really change the confusion at the usage-site: to me it seems it's mostly the people using extension methods which bear the brunt of the confusion, not the people defining them.
    • Non-extension method implicit conversions (implicit constructors in the companion object) can be used very heavily with no usage-site confusion at all (e.g. see most of the com.lihaoyi libraries: requests-scala, os-lib, fastparse, sourcecode, etc., nobody has complained in years about those implicit conversions!).
  2. How does changing implicit val or implicit def for witness help tooling? How can tools give a precise list of options, when witnesses are similarly context-dependent as implicits? How does it help make error messages of deeply recursive failures more specific? It's obvious that more progress is good; it's less obvious how this proposal constitutes progress.

  3. How is witness foo less minimal than implicit def foo or implicit val foo? Certainly the former has fewer tokens and fewer characters. Would letting people define things as simply implicit foo solve the problem?

  4. In terms of syntactic bikeshedding, it seems to me like a sideways change.

    • The current weirdness of "implicit only occurs once per-parameter list" is a non-issue for me

    • The "definition site doesn't align with callsite" issue can be solved trivially by making people put an implicit keyword at the callsite argument list too.

    • The overloading of with and for to mean something completely unrelated to what they currently mean is a negative to me.

    • I can't understand what the dependency injection example is trying to do even after reading it and the surrounding text several times.

    • I would like to offer #1260 (comment) as an alternative syntax that I believe would be more intuitive, and could be fleshed out more completely if necessary.

There's some interesting bits and pieces in here (e.g. anonymous witnesses/implicits) but overall I think the proposal doesn't make a sufficiently good case about why exactly this, and not something else, is the correct-enough solution for a big-enough problem that is worthy of the huge syntactic change over the status quo.

For example, while the Convertible_List_List_witness example looks like a big improvement in clarity for the new system, anonymous implicits can be implemented on their own as their own thing, and so can something like C# 8's Target-Typed New Expressions, and together would give you all the benefits of clarity and conciseness with only tiny changes:

implicit def [From, To](implicit c: Convertible[From, To]): Convertible[List[From], List[To]] = new {
  def convert (this x: List[From]): List[To] = x.map(c.convert)
}

Or for the Ord example:

implicit val IntOrd: Ord[Int] = new {
  def compareTo(this x: Int)(y: Int) =
    if (x < y) -1 else if (x > y) +1 else 0
}
implicit def ListOrd[T: Ord]: Ord[List[T]] = new {
  def compareTo(this xs: List[T])(ys: List[T]): Int = (xs, ys) match {
    case (Nil, Nil) => 0
    case (Nil, _) => -1
    case (_, Nil) => +1
    case (x :: xs1, y :: ys1) =>
      val fst = x.compareTo(y)
      if (fst != 0) fst else xs1.compareTo(ys1)
  }
}

If we could find similar ways to get most of the other benefits of this proposal with small, standalone changes that are themselves valuable on their own, to me that would be preferable over a big-bang rewrite-everything migration. I suspect we can.

Some ideas to make the proposal better:

  • List out explicitly how the initial problems laid out get solved by this proposal: how does this affect tooling? How does changing implicit def for witness make the syntax less minimal?

  • Can we separate the syntactic changes from the semantic changes? e.g. can we get the sweet tooling-side improvements to deep recursive implicit failures without overhauling a bunch of irrelevant syntax?

  • Perhaps define the proposal in a series of step-by-step delta change from the status quo, rather than a "here is a whole new world and how it works". That would make it a lot easier to properly analyze each step without getting overwhelmed in a sea of unrelated changes

@fanf

This comment has been minimized.

fanf commented Nov 17, 2018

I understand @lihaoyi feedbacks, and I believe his stated 3 points are a good trail to help the proposal keep going forward.
I would still let know that from my corner of social network (https://twitter.com/fanf42/status/1063573841014923265), the general feedbacks are goods, some are even almost hyporbolic about the goodness of the proposal. The main concerns are the same than the one pointed by @lihaoyi: the DI part is more a burden than an asset, the value proposition past the syntax change is not sufficiently apparent, the benefits to help with the tools not made clear - and of course, the migration path seems daunting. All in all extremelly positive :)

@Jasper-M

This comment has been minimized.

Contributor

Jasper-M commented Nov 17, 2018

@fanf A lot of the positive feedback seems to be directed at #5114 on which this PR is based. I don't really see the benefits of the extra witness stuff over #5114 with implicit.

@odersky

This comment has been minimized.

Contributor

odersky commented Nov 17, 2018

I dropped the dependency injection section. It is now mostly confusing and does not convey a new point. There was a point once to it, in an earlier design (before b86e91d) where the two roles of an implicit parameters of (1) taking implicit arguments, and (2) being itself a witness for further implicits were separated. Then the example was much nicer. It was left in, to show that it's in principle still possible to do this even if the two parameter roles are not separated. But without that earlier design
it misses context.

@jeremyrsmith

This comment has been minimized.

jeremyrsmith commented Nov 17, 2018

Just wanted to say, as someone who "gets" implicits, this is much harder for me to understand than implicits. The only problem with implicits (problem being that they are difficult for newcomers to grasp) is that the term is applied to two different things – implicit conversions (I call these "non-explicit conversions") and actual implicits (facts, derivations, and bounds).

Newcomers conflate these things and end up thinking all implicits are bad, because implicit conversions are bad (I've heard so many people call typeclasses "implicit conversions"). But it is the actual (good) implicits that are getting hosed here. It seems backwards to me that the good parts have to change rather than the bad parts.

Wouldn't it be simpler to just change the syntax for implicit conversions? Like maybe you have to say implicit conversion fooToBar (foo: Foo): Bar instead of implicit def, to make it clearer? I don't know what the syntax would be, but I don't look forward to having to re-learn implicits as witnesses instead (not to mention the confusion with shapeless' Witness, though I suppose shapeless will have a very different form when it gets to dotty in any case).

Another thought: Just because something takes a little bit of understanding doesn't make it bad. I think a lot of the hysteria (for lack of a better word) around implicits in Scala is misplaced and is better solved with education than with trying to radically redesign it. Once people learn about it they generally come to understand that it's a really elegant mechanism for abstracting over many elements of the type system in terms of "theorem proving" (in a lightweight sense). But without removing that power, or removing the elegant abstraction, you can't change the fact that it requires a little bit of a paradigm shift for a career Java programmer to understand. And they have to want to do that – some people won't, and that ought to be OK.

@jeremyrsmith

This comment has been minimized.

jeremyrsmith commented Nov 17, 2018

And one more comment, sorry: Solving some of the ergonomics around implicits would get a lot further than changing the name to escape its baggage. A better story around user-friendly error reporting for failed proofs would solve 80% of the complaints. A story around coherence and orphans would solve probably a further 15% (if such a story is possible). The remaining tail of complaints typically come from people who complain about "magic" (but then will happily use runtime-reflection driven dependency injection stuff like Guice...)

@odersky odersky force-pushed the dotty-staging:add-witness branch from 7c726d7 to 8a77d8c Nov 17, 2018

@mdedetrich

This comment has been minimized.

mdedetrich commented Nov 17, 2018

I am going to have to go through this in a bit more detail but I am having a hard time getting convinced about what the improvement witness's actually bring, kind of echoing @lihaoyi points but I am have more of my own which I am going to list.

With your example of the apply being mixed up with the implicit parameter invocation in

def currentMap(implicit ctx: Context): Map[String, Int]

This can easily be solved by simply stating that all apply's must use the non implicit parameters. If you want to then implicitly supply a parameter, you can simply do something like (in the above example)

currentMap.implicitly(someContext)
// or this version which can also solve the currying issues that Scala has with implicits
currentMap("someKey").implicitly(someContext)

This would allow you to do

implicit val ctx: Context = ???
currentMap("someKey")

As desired.

The proposed syntax also has a lot of consistency issues, I am actually for more making the current implicit syntax more consistent, i.e. something like #1260 (comment)

Regarding dependency injection, this can already be done naturally with implicits and traits currently.
i.e. you can define your dependencies like this (for your typical standard backend app)

class UserClient(implicit connectionDetails: ConnectionDetails, scheduler: Scheduler) {
   ...
}
class AddressClient(implicit connectionDetails: ConnectionDetails, scheduler: Scheduler) {
  ...
}

trait ServicesConfig {
   lazy val userClient = ???
   lazy val addressClient = ???
}

trait DatabaseConfig {
   implicit lazy val connectionDetails = ConnectionDetails("localhost",8080)
}

trait SchedulerConfig {
   implicit lazy val scheduler = Scheduler(executionContext)
}

object Main extends App with ServicesConfig with DatabaseConfig with SchedulerConfig {
   logger.info("Initialized")
}

And you can do the thing that you actually want to do with dependency injection, which is replacing specific "nodes" of your dependency graph, i.e.

lazy val testApp = new ServicesConfig with DatabaseConfig with SchedulerConfig {
   override lazy val userClient = MockUserClient(....)
}

EDIT: I realized that the DI section was dropped but I will still leave this here to demonstrate that the current implicit's solve the DI problem fine.

Also in regards to syntax, we really shouldn't overload the meanings of keywords. Although claiming that Scala has fewer keywords than other languages is a notable goal it shouldn't be at the cost of giving keywords overloaded meanings.

For example, for is typically only ever used with for comprehensions. Using the syntax this way for witness's is confusing at best.

Note I am not really saying that the witness is pointless, but I would rather not throw away the baby with the bathwater and fix the current issues we have with implicit's and then evaluate if we need witness and how much value it adds. For example claiming that current implicits are problematic because the compiler gives bad information when implicits are missing, honestly this to me sounds like a scapegoat. There is nothing stopping the compiler from giving detailed warnings proviso performance issues and if thats the case I don't see how witness's would be any better here.

Also I would consider removing implicit conversions (which iiirc are currently deprecated) or severely limit them, this is the only concrete issue I have seen with implicit abuse in the Scala ecosystem

@odersky

This comment has been minimized.

Contributor

odersky commented Nov 17, 2018

Just wanted to say, as someone who "gets" implicits, this is much harder for me to understand than implicits.

That's a good point. Witnesses are at best a "sideways" move for someone who gets implicits, which is everyone in this discussion. But my focus is, what about somebody who does not get implicits because they are new to Scala? Programs should express meaning and intent. Implicits express mechanism instead. An eye opener for me was when I worked on extension methods (which are by themselves a huge improvement for expressing typeclasses) and was left with

implicit object Foo {
  def extMethod(this x: T): U
}

Really? That seemed like a sore thumb sticking out. implicit object is such a weird incantation to make an extension method available! This was an eye-opener: Scala uses the modifier "implicit" in many situations which are positively misleading to a newcomer. An implicit object or an implicit def is just as explicit as a regular object or method. The fact that implicit means it's available as an argument to an implicit parameter is mystifying to someone who is not already familiar with the concept. I like to think of Scala generally as a clear and readable language, but the use of implicit goes counter to that.

Doing some language archeology, the word implicit was originally used in Scala to define an implicit conversion. There it makes sense. You define a method that can be inserted implicitly. A second step (around 2006) was that implicit resolution for conversions could also be used for implicit parameters which could model type classes. I had a talk "poor man's type classes" at a WG 2.8 meeting in that year. Since it seemed so close, we re-used the implicit modifier for parameters and type class instances.

Fast forward 12 years. By now, most uses of implicit conversions are frowned upon and the language will make it ever harder to define and use them. But the word implicit sticks around, and is getting attached to more and more concepts (i.e. implicit classes, implicit function types). For someone who just discovers Scala, this must be very confusing. I argue we should stop it, and 3.0 is a good occasion to do so.

It's really easy to rewrite automatically, so I am not worried about migration at all. In terms of breaking code I am not worried either, since old implicits will be kept around for a while, then be deprecated for a while, before they would be finally removed in 3.2 or 3.3. It's true that a lot of books and other material will have to be rewritten. But let's face it: Much of the material in these books dealing with implicits would benefit from a rewrite. Many treatments that I have read (or written!) start with implicit conversions, which by now is quite cringe-worthy!

@odersky

This comment has been minimized.

Contributor

odersky commented Nov 17, 2018

@lihaoyi

Is defining too many implicit conversions really a problem right now?

The feedback I get from others indicates this. It seems to be the #1 problem with implicits.

As a library user I'd argue that the most problematic usage patterns are people defining too many extension methods, which this proposal seems to encourage.

This is interesting. I did not see it that way. There are languages (i.e. Rust, and, to a lesser degree Swift) built around extension methods to do everything! Why do you think they are a problem? Is it that a missing import causes obscure errors? We could try to come up with solutions that address this problem separately.

Non-extension method implicit conversions (implicit constructors in the companion object) can be used very heavily with no usage-site confusion at all (e.g. see most of the com.lihaoyi libraries: requests-scala, os-lib, fastparse, sourcecode, etc., nobody has complained in years about those implicit conversions!).

True. That seems to be the one usage of implicit conversions that carries its weight: Define them with that target type to which they convert.

How does changing implicit val or implicit def for witness help tooling?

It's completely orthogonal. Witnesses are a fairly simple syntactic and type transformation. The most important thing is that they force alignment of parameter definitions and arguments.

How is witness foo less minimal than implicit def foo or implicit val foo?

  • it's shorter
  • it's just one word for the concept rather than two which have to be combined

This adds up! Meaning instead of mechanism...

Certainly the former has fewer tokens and fewer characters. Would letting people define things as simply implicit foo solve the problem?

No, since it is not implicit. Implicit is a misnomer.

I have to jump on a plane, so I have to leave it at that.

@arturopala

This comment has been minimized.

arturopala commented Nov 17, 2018

Maybe implicit is a misnomer, but IMHO it sounds correct for implicit conversions and implicit parameters. I agree it sounds weird when declaring implicit object or class, so maybe only for the later use case, we could find the right keyword, e.g. particular, evident, explicit or singular?

As for with keyword to supply specify implicit arguments, does it express the intent? It sounds somewhat confusing to me in this context, whereas explicitly would be clear and unambiguous.

@lihaoyi

This comment has been minimized.

Contributor

lihaoyi commented Nov 18, 2018

This is interesting. I did not see it that way. There are languages (i.e. Rust, and, to a lesser degree Swift) built around extension methods to do everything! Why do you think they are a problem? Is it that a missing import causes obscure errors? We could try to come up with solutions that address this problem separately.

I suppose Go also can be thought of as using extension methods for everything, and they seem to not have a problem.

The only difference i can think of is that in Go/Rust/Swift, extension methods aren't "first class" IIRC: you can only define them top-level and import them from top-level packages, and can't do clever tricks like defining them in a trait, inheriting them in a val, pass the val around as and them importing them from the val.

Scala is one of the only language which allows this, because most languages don't let you import from arbirary values. In fact, Scala explicitly prohibits defining extension methods top-level in a file like most other languages! Perhaps that is what causes confusion in Scala?

it's shorter
it's just one word for the concept rather than two which have to be combined

This seems like using witness foo is more minimal than implicit val foo. However, you stated up top that the problem with implicit val foo was that it was too minimal already! Making something that is too minimal even more minimal doesn't seem like solving the problem you stated.

No, since it is not implicit. Implicit is a misnomer.

I'd argue that the idea of implicits in programming are actually very well known, are present in several very popular languages by that exact name:

  • C++ has implicit constructors
  • C# has implicit constructors
  • Most languages have threadlocals which are used in a similar way to Scala implicit parameters, just less safely, which can also be described as "implicit"

Having parameters or conversions be implicits is not that foreign a concept.

On the other hand, the idea of a witness in programming is almost unheard of: perhaps discussed in academic or advanced theorem proving scenarios, but nowhere else. People generally do not talk about passing witness or evidence values around, even people using Scala!

Consider Googling programming implicit vs programming witness:

programming implicit

  • "An explicit conversion is one which has to be stated explicitly; an implicit version is one that can be used implicitly, i.e. without the code having to state it" - dictionary - explicit and implicit c# - Stack Overflow
  • What is an implicit variable?
  • What is an implicit function in C++?
  • What is an implicit casting?
  • What is an implicit declaration in VB?

programming witness

  • Semantic Compression (blog post about the video game The Witness)
  • A note about programming design - The Witness
  • Witness wednesday on How to Program - The Witness
  • Testing the Jai Compiler - The Witness
  • Jonathan Blow "C++ is a weird mess": The Witness developer on his new programming language

By replacing "implicit" with "witness", to the vast majority of programmers you are replacing a probably-seen-before-in-similar-concept key word with the name of a video game. You can't get much more misnomer than that!


It's worth bringing up the question: what is people's real problem with implicits in Scala? People get by just fine with extension methods in Rust, Go or Swift, and people get by just fine with implicit conversions/constructors in C++ or C#: maybe you don't like them or use them all the time, but they don't cause the average person too much pain using the language. You can very much use Scala like C#: with implicit constructors to save on boilerplate and extension methods to tweak the syntax, once in a while. Why then do implicit parameters and implicit conversions get a bad rap in Scala?

I'd argue the problem isn't so much implicits themselves, but the usage of implicits in the community. This starts, first and foremost, in the standard library: collections and CanBuildFrom are code that everyone uses, everyone jumps-to-definition, and either learns from or recoils away (both of which are bad!). The core/early libraries also help set the culture: with Databinder Dispatch and it's operators, Scalatest and Specs2 with their zoo of inherited extension methods, SBT (which also contributed scala.sys.process), Slick and it's path-dependent inherited implicit-parameter-taking implicit-conversions, scala-pickling's error-prone top-level implicit imports, and others all setting the standard for "this is how Scala is and this is how Scala should be written". Even if you ban implicits in user-code entirely, e.g. with compiler flags, they still end up suffering from confusing implicits in the standard and third-party libraries.

To solve this, the solution would be to build a core ecosystem that uses implicits in a "good way", get it out there into the community, popularize it, and get people to build upon that both for their own happiness and as example code to learn from:

  • Can we write a HTTP library that uses implicits in a "good" way?
  • A subprocess library?
  • A filesystem library?
  • A build tool?
  • A unit testing framework?
  • A serialization library?
  • A http client? Server framework?

If we can, then at least we'll have figured out what the "good" ways of using implicits are, and therefore we would like to encourage via language tweaks. And if we don't even know what it means to build "good" implicit-using library code ourselves, there's no way we'd be able to make the right decision on how to tweak the language to make other people improve their usage of implicits!

@arnolddevos

This comment has been minimized.

arnolddevos commented Nov 18, 2018

It's worth bringing up the question: what is people's real problem with implicits in Scala?

The problem is implicit scope. I am surprised that is not mentioned more prominently in these comments.

Implicit scope is hard to visualise. For the user of an implicit there always seems to be one too many places to look for the definition.

For the author there is always a niggling concern that the implicit might not be available when it should or that an implicit might escape and affect something it shouldn't.

Different libraries have different conventions that the user should follow for bringing necessary implicits into scope.

I don't think users of rust traits, for example, have these mental speed bumps.

```
Context bounds in witness definitions also translate to implicit parameters, and therefore they can be represented alternatively as with clauses. For instance, here is an equivalent definition of the `ListOrd` witness:
```scala
witness ListOrd[T] with (ord: Ord[T]) for List[Ord[T]] { ... }

This comment has been minimized.

@katy-p

katy-p Nov 18, 2018

I believe it should be Ord[List[T]] instead of List[Ord[T]].

An underscore ‘_’ can be used as the name of a required witness, if that witness does not
need to be referred to directly. For instance, the last `ListOrd` witness could also have been written like this:
```scala
witness ListOrd[T] with (_: Ord[T]) for List[Ord[T]] { ... }

This comment has been minimized.

@katy-p

katy-p Nov 18, 2018

I believe it should be Ord[List[T]] instead of List[Ord[T]]. (save as above)

@LPTK

This comment has been minimized.

LPTK commented Nov 18, 2018

I agree that the word "witness" seems inadequate, outside of academia.

Why not "capability"? After all, that's what implicit parameters really are.

It works well with extension methods:

capability Foo {
  def foo(this x: T): U
}

I'd say that "capability" is also a better name than "type class" (which is quite terrible, because "class" already means something else in Scala). The following makes perfect sense to me:

def foo(x: T)(capability s: Show[T]) = ...

So first-class capabilities are the distinguishing feature of Scala, not implicits, even though capabilities are usually (but not always!) passed around implicitly. Other languages have various forms of implicits, but no mainstream language has first-class capabilities.

Also, as @arnolddevos says, it does seem like implicit scope (first-class capability scope) seems to be the most problematic thing that people complain about.

@jenshalm

This comment has been minimized.

jenshalm commented Nov 18, 2018

@odersky I feel like this is going in the right direction, I completely agree with the problem analysis of what is inconsistent and confusing about current implicit syntax and think the new proposal provides a neater syntax that describes the intent more clearly.

The sole concern for me, which seems to be shared by some commenters here, is the choice of name (witness), which I find mildly confusing. I might be missing something from common cs terminology, but for me a witness is something rather passive, where the action had been somewhere else and the witness merely testifies that, e.g. in an implicit witness parameter that just testifies that A is indeed a subtype of B. But in this case a witness does not just testify that an instance for type class F[A] exists, it actually does provide the implementation for it (apart from the alias examples). Maybe this is semantic hair-splitting, I was just getting a bit worried about the blank stare I get from newbies in the future when I tell them: "you just need a witness in scope". :-)

My mind is in a lazy Sunday haze so I sadly do not have a counter-proposal for a name and instead would just like to ask: could you describe why you think this is a good choice for a name? Have you considered any alternative names that you later discarded? Are there examples from other languages where this term is used in a similar way?

The only other question is: from the four reasons you give in the motivation section, three are addressed by the proposal in a pretty obvious way. The only exception seems to be tooling. If there is an obvious way this proposal makes smarter tooling easier, it might be nice to explicitly describe the advantage.

@odersky

This comment has been minimized.

Contributor

odersky commented Nov 18, 2018

@jenshalm I arrived at "witness" after a some iterations. I had "extension"and "instance" before, but both looked too generic. I don't know. Judging from Twitter, people doing type classes like "witness" and it seems quite clear to them.

Other languages are not much better: impl, instance, extension are all fairly generic and ad-hoc names. What I like is that witnesses enforce the mindset that types are properties and term instances, being witnesses of types, show that the property is implementable.

@jducoeur

This comment has been minimized.

Contributor

jducoeur commented Nov 18, 2018

(Still need to read properly into the proposal, but based mainly on the conversation --)

What I like is that witnesses enforce the mindset that types are properties and term instances, being witnesses of types, show that the property is implementable.

I can sort of see that, but honestly -- I'm not at all sure how to teach that to novice Scala programmers in a way that they will actually understand.

I kind of like "implicit" because I've figured out over the years how to teach it: I describe a "cloud" of implicit definitions around the code. The implicit keyword either adds stuff to that cloud or summons stuff from. Folks generally seem to get the idea from that description, rough and ready though it is.

This comment kind of worries me in that respect. The concepts of "type" vs. "term" rarely come up in introductory Scala courses -- I don't think we even get into that in our advanced ones. And the notion of program-as-proof rarely comes up. On-the-ground programmers generally don't care about this stuff, and I avoid it rigorously while teaching -- it's the sort of topic that drives many folks to think of Scala as "too hard"...

@senia-psm

This comment has been minimized.

Contributor

senia-psm commented Nov 18, 2018

Can one create a class with context parameters?
Like this:

class Context
object Context extends Context

class MyClass(name: String) with (ctx: Context)

In this case it's quite hard to tell what new A with B means:

new MyClass("name") with Context // looks like inheritance
@odersky

This comment has been minimized.

Contributor

odersky commented Nov 18, 2018

new MyClass("name") with Context // looks like inheritance

I would be in favor of dropping new A with B. If you need a class with more than one parent, name it! This would us allow to retirewith for inheritance altogether. Since with no longer means intersection type, using it for inheritance is kind of pointless. A comma would do just as fine, would be shorter and more readable.

@odersky

This comment has been minimized.

Contributor

odersky commented Nov 18, 2018

I can sort of see that, but honestly -- I'm not at all sure how to teach that to novice Scala programmers in a way that they will actually understand.

I believe implicits or witnesses is not a subject you want to teach to a novice programmer anyway!

@jducoeur

This comment has been minimized.

Contributor

jducoeur commented Nov 18, 2018

I believe implicits or witnesses is not a subject you want to teach to a novice programmer anyway!

Honestly, I don't think there's much choice. Even a fairly straightforward concept like Ordering requires basic typeclasses, and doing anything with Future requires explaining implicit ExecutionContext, so having the slightest clue what's going on there requires explaining implicit parameters.

No, it's not something I teach in the first few days. But in practice, as soon as an engineer hits a non-trivial codebase they will hit implicits, and the more curious ones (especially the experienced programmers -- I'm not talking about new programmers, I'm talking about new Scala programmers) get really annoyed if you try to brush them off as something they shouldn't worry their little heads about.

I don't go deeply into implicits at that point, mind. But being able to provide some intuition about what is going on, in terms that they can grok, is pretty important fairly early on, in my experience...

@acjay

This comment has been minimized.

acjay commented Nov 19, 2018

I'm a big supporter of rethinking implicit. As someone who has had to mentor new Scala practitioners, implicit is one of the biggest hurdles. The semantic mismatch between the actual definition of the word and how it works in Scala is a big part of this. Even now, there are advanced uses I can't wrap my head around because of the divergence between the literal code and its intended semantics. If accessibility to newcomers is a top priority, even a lateral change could make a big k improvement.

I think the parameter list representation of implicits is indeed one of the hardest parts to understand, and it's also one of the hardest usages of the keyword to avoid in basic real-world Scala programming. Consider the pervasive need to pass ExecutionContext if you're using standard library Future or ActorSystem if you're using Akka. It scares the hell of of people on Day 1 and confirms their suspicion that the language is complex. Any reform should be measured against these use cases just as much as against the more advanced uses of implicit.

I disagree with the idea that implicit is familiar to users of other languages. My experience elsewhere is almost totally limited to implicit conversions, which is often not what people do in idiomatic Scala. I'd argue that this association actually makes Scala's system more difficult to grasp, because the analogy doesn't go very far.

capability seems like the best suggestion so far, but I can also imagine context being a good descriptor. The latter is reminiscent of React, which has a concept called context, which is rather like a really simple version of the implicit scope mechanism.

Related to that question, I do think it could be very helpful to separate the conveyance of implicits down the call stack from their summoning. For conveyance, complete anonymity is preferable. For summoning, you don't necessarily need to convert the implicit any further. It could be very helpful to separate these concerns in a way that could be combined, but only if necessary. The context bound sugar does a great job of this in that particular use case of implicit.

As an aside, React's context is actually more implicit, in the literal sense, than Scala implicits, because you don't need to declare it anywhere between the provision and consumption, unlike Scala's use of implicit parameter lists. Would this be possible to achieve? In this case, conveyance could literally be implicit.

@odersky

This comment has been minimized.

Contributor

odersky commented Nov 19, 2018

@jducoeur @acjay @LPTK I agree implicit parameters come up immediately when teaching some libraries. One of the first times I mention implicits in my courses is ExecutionContext. And it is confusing to make them look like normal parameters, which is another reason to change the syntax to with.

Implicit objects and classes? I believe that's comparatively rarer, and it's a harder concept to grasp, for the following reason:

Type classes are a second order concept - they are types that describe properties of other types, which may or may not hold depending on context. I.e.

  • Ord[T]: T has an ordering
  • Monoid[T]: T has "combine" and "unit" operations
  • CanBuildFrom[From, Elem, To]: One can construct a collection of type To with elements of type Elem from a collection of type From.

That idea takes some getting used to but I believe it is essential to be clear about this if one wants to teach type classes. My earlier, abandoned attempt at extensions was in part motivated by the wish to delay the introduction of type classes by modelling context dependent extensions instead. That approach introduced other complexities; in particular it required additional machinery for talking about non-extension methods such as unit in Monoid. The idea of, first, the current version of extension methods and, subsequently, witnesses came when realizing that ultimately the language becomes simpler if we embrace type classes rather than hiding them.

But once one introduces type classes as properties of types, I believe witness is a descriptive term for type class instances. Is T ordered? It depends on the context. One needs a witness to establish that fact.

capability is an interesting alternative, which has a lot going for it. We often use implicit parameters to represent capabilities. That's already handled with with but capability also works well for something like the global ExecutionContext - that's a system wide capability to execute something. It's less obvious for Ord[Int] - yes, you can say, here's a capability to compare Ints, but I prefer to start with the property "Int is ordered" and say here's a "manifestation" or "witness" for that fact.

The other problem with capability is that defining a "global" capability like we do with a type class instance is an anti-pattern. The capability security model explicitly disallows this.

The last (maybe decisive?) problem is that it is longer than "witness" and has 5 syllables instead of two. So there's that...

@odersky

This comment has been minimized.

Contributor

odersky commented Nov 19, 2018

The other thing I like about witness is that it is an easy to grasp description for Curry-Howard: "Types are properties and terms are proofs". Sure, that's academic, but I believe it will become a lot more industrially important in the future.

odersky added some commits Oct 29, 2018

Reworked proposal
 - drop distinction between implicitly passed and witness parameters
 - drop implicit as a modifier
 - use `with` instead of prefix `.` as connective for context parameters and arguments
 - use `|=>` for implicit function types and implicit closures

# Conflicts:
#	docs/docs/internals/syntax.md
Parsing of contextual (`with`) parameters and arguments
# Conflicts:
#	compiler/src/dotty/tools/dotc/parsing/Parsers.scala
#	docs/docs/internals/syntax.md
Changes to implicit function types
 - Introduce `|=>` syntax for implicit function types and implicit closures
 - Drop old `implicit A => B` syntax
 - Make implicit function types contextual, i.e. parameters have to be passed
   with `with`.

# Conflicts:
#	tests/pos/reference/witnesses.scala
Fix dependent function type building
Need to record function modifiers in parameters, or they will get
lost in pickling. Fixes failing pickling test pos/depfuntype.scala.
Flesh out migration discussion
+ fix tests added by rebase
Revert alternative infix syntax for extension methods
That one was slipped in by accident.
Drop dependency injection section
It is now mostly confusing and does not convey a new point. There was a point once to it,
in an earlier design where the two roles of an implicit parameters were separated. Then
the example was much nicer. It was left in, to show that it's in principle still possible
to do this even if the two parameter roles are not separated. But without that discussion
it misses context.
Handle right-associative extension operators
# Conflicts:
#	compiler/src/dotty/tools/dotc/parsing/Parsers.scala
Support witness defs with () parameters
... and disallow the `: => T` syntax.
Remove dead code in Parser
# Conflicts:
#	compiler/src/dotty/tools/dotc/parsing/Parsers.scala
Make `instance` a reserved word
# Conflicts:
#	compiler/src/dotty/tools/dotc/transform/Splicer.scala
witness -> instance
# Conflicts:
#	compiler/src/dotty/tools/dotc/parsing/Parsers.scala
Revert: Dotty docs for "this as modifier" scheme
(reverted from commit cf68e01)
Handle positions in extension methods

1. Take extension methods into account when checking positions.

2. Fix position of empty type tree in method definition

Without this measure, the empty type tree gets the end position of
the preceding element. For an extension method with type parameters
this is the extension parameter list, but it should be the type parameters.
Fixing the position explicitly fixes the problem.
Update tests to use prefix parameter syntax for extension methods

@odersky odersky force-pushed the dotty-staging:add-witness branch from 0b41685 to 1ddcc2c Dec 14, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment