New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SIP-NN Inline/meta #567

Merged
merged 4 commits into from Sep 26, 2016

Conversation

Projects
None yet
9 participants
@xeno-by
Member

xeno-by commented Sep 9, 2016

tl;dr. This Scala Improvement Proposal suggests a new macro system for inclusion into Scala as a stable, non-experimental part of the language. It is based on our experience with the current, experimental macro system available since Scala 2.10.

Document:

SIP-NN Inline/meta

Discussion:

Pending changes:

  • Add warning, error and possibly other APIs for diagnostic messages.
  • Add meta types as a solution to whiteboxity (https://gitter.im/scalameta/sips?at=57bd9a1a87f779f06923bfc2).
  • Provide a repository that implements the example from the Intuition section using old-style and new-style macros.
  • Update the inline part of the SIP with findings of a Dotty prototype (lampepfl/dotty#1492): 1) do we allow return in inline defs?, 2) reconsider overriding, 3) reconsider eta expansion, etc.
  • Clarify the notion of compile-time constants (https://twitter.com/xeno_by/status/767745037719535616).
  • Explain that inline defs without meta expressions inside are not affected by the separate compilation restriction.
  • Clarify the interaction with structural types. In particular, can an instance of a class with an inline method be cast to a structural type with the same method, but without an inline tag?
  • Provide more details about how inline expansion interacts with private members used in bodies of inline methods.
  • Can we really supersede @inline with inline given that @inline can be applied to virtual methods (https://gitter.im/scalameta/sips?at=57bc1368cd00bdff6e70f212)?
  • Clarify whether inlinity changes overloading. E.g. can we overload on inlinity?
  • Reconsider rule 1 of inline reduction. How does it interact with side effects in bodies of inline values?
  • Explicitly explain what blackboxity and whiteboxity mean.
  • Provide an example that illustrates what's wrong with not using decodedName.
  • Fix the typos in Intuition that refer to Select nodes as Map.
  • Provide concrete examples of how inline expansion works with by-value and by-name parameters.
  • Fix a typo in macros in Intuition (https://github.com/scala/scala.github.com/pull/567/files/3520d5c761abfdfeff66d396456791971795f6af#r78260027)
  • Consider applying scala code highlighting to code examples
  • Split the SIP into two parts
@acdenhartog

This comment has been minimized.

Show comment
Hide comment
@acdenhartog

acdenhartog May 13, 2016

Is this SIP being discussed anywhere?

First of all regarding inline def async[T](x: => T): T = meta { ... },

There is already the @inline annotation, what is the advantage of adding a new keyword instead of modifying how the compiler handles that existing annotation? The compiler would only have to look for a meta block to choose the correct behavior of @inline.

I do not think this could be confusing since the effects of @inline and this inline keyword are conceptually identical. There are only some extra rules regarding overrides, erasure, and reflective calls. I think having both could be more confusing to newcomers.

As for that other inline thing with val or method parameters, what is the point of doing that? And what if people do it on a var? That just seems like a nice syntax for creating bugs.

And this bit:

The previous rule of regarding a final val with no explicit type as a compile-time constant (inherited from Java) is dropped. Instead we write such vals now as inline. Value and method definitions labeled inline are effectively final;

That is simply using a new keyword to replace something that an old one already does. Why?

Eta expansion of inline methods is prohibited.

Technically yes, but one could just write: (a,b) => inlinedMethod(a,b) which is the same as eta expansion from a users point of view. I think inlinedMethod(_,_) syntax would have to be supported too just for the sake of consistency. And don't forget that type parameters must always be bound at the point of eta expansion, so that meta block would just receive those normally under eta expansion.

We believe that, with additional effort, inline methods and inline parameters can be used to express partial evaluation of methods like pow in the introduction,

I have no idea how that is supposed to work. pow is not even using meta, so that is just a regular inline. And pow tail-recursive, so its inline would just be a while-loop or a stack-overflow in the compiler. How is inline even supposed to work with normally recursive methods?

I can see it working in the following rather contrived example:

  unWraps(Option(Option(Option(None))))
  inline def unWraps[T<:Option[T]](it:Option[T]): Unit =
    it match {
      case None => ()
      case Some(deeper) =>
        unWraps(deeper)
        print("g")
    }

Don't get me wrong, I am all for continuations, async, and type-level hackery; I just have no idea how this inline, without the use of meta, is supposed to help with that. Is there really a compelling use case for inline without it being attached to a macro?

So in general, there is one part of this SIP that I like: inline def method[T]: T = meta { ... }; and I do appreciate choice of the word inline over macro; but I do not see what there is to be gained from the rest of this SIP.

Although, what would be the advantage of writing both inline and meta instead of only macro in front of a block? That would be somewhat DRYer. Perhaps it is because you do not wish for an inflexible binding onto scala.meta as the only macro expansion system, so someone might later write inline def method[T]: T = ACMEmacro { ... }, but that should be easy to solve some other way.

Lastly, there is a solution to the whitebox macro issue: just write your entire program as one gigantic collection of quasiquotes. One meta block to rule them all!!! One typer to find them, one inline to bring them all and in the Unit bind them!

Okay, that is more than enough of me...

Edit: by the way, is there a prototype of this implemented anywhere? I suppose paradise annotations should be powerful enough to create @protoinline def method[T]: T = meta { ... } assuming that meta itself has already been implemented somewhere...

acdenhartog commented on 3b8056d May 13, 2016

Is this SIP being discussed anywhere?

First of all regarding inline def async[T](x: => T): T = meta { ... },

There is already the @inline annotation, what is the advantage of adding a new keyword instead of modifying how the compiler handles that existing annotation? The compiler would only have to look for a meta block to choose the correct behavior of @inline.

I do not think this could be confusing since the effects of @inline and this inline keyword are conceptually identical. There are only some extra rules regarding overrides, erasure, and reflective calls. I think having both could be more confusing to newcomers.

As for that other inline thing with val or method parameters, what is the point of doing that? And what if people do it on a var? That just seems like a nice syntax for creating bugs.

And this bit:

The previous rule of regarding a final val with no explicit type as a compile-time constant (inherited from Java) is dropped. Instead we write such vals now as inline. Value and method definitions labeled inline are effectively final;

That is simply using a new keyword to replace something that an old one already does. Why?

Eta expansion of inline methods is prohibited.

Technically yes, but one could just write: (a,b) => inlinedMethod(a,b) which is the same as eta expansion from a users point of view. I think inlinedMethod(_,_) syntax would have to be supported too just for the sake of consistency. And don't forget that type parameters must always be bound at the point of eta expansion, so that meta block would just receive those normally under eta expansion.

We believe that, with additional effort, inline methods and inline parameters can be used to express partial evaluation of methods like pow in the introduction,

I have no idea how that is supposed to work. pow is not even using meta, so that is just a regular inline. And pow tail-recursive, so its inline would just be a while-loop or a stack-overflow in the compiler. How is inline even supposed to work with normally recursive methods?

I can see it working in the following rather contrived example:

  unWraps(Option(Option(Option(None))))
  inline def unWraps[T<:Option[T]](it:Option[T]): Unit =
    it match {
      case None => ()
      case Some(deeper) =>
        unWraps(deeper)
        print("g")
    }

Don't get me wrong, I am all for continuations, async, and type-level hackery; I just have no idea how this inline, without the use of meta, is supposed to help with that. Is there really a compelling use case for inline without it being attached to a macro?

So in general, there is one part of this SIP that I like: inline def method[T]: T = meta { ... }; and I do appreciate choice of the word inline over macro; but I do not see what there is to be gained from the rest of this SIP.

Although, what would be the advantage of writing both inline and meta instead of only macro in front of a block? That would be somewhat DRYer. Perhaps it is because you do not wish for an inflexible binding onto scala.meta as the only macro expansion system, so someone might later write inline def method[T]: T = ACMEmacro { ... }, but that should be easy to solve some other way.

Lastly, there is a solution to the whitebox macro issue: just write your entire program as one gigantic collection of quasiquotes. One meta block to rule them all!!! One typer to find them, one inline to bring them all and in the Unit bind them!

Okay, that is more than enough of me...

Edit: by the way, is there a prototype of this implemented anywhere? I suppose paradise annotations should be powerful enough to create @protoinline def method[T]: T = meta { ... } assuming that meta itself has already been implemented somewhere...

This comment has been minimized.

Show comment
Hide comment
@xeno-by

xeno-by May 13, 2016

Member

Hello @acdenhartog, thanks for writing! We're all travelling this week, so communication may be delayed. In the coming days, I'll arrange a channel to discuss the SIP and invite everyone to join. I'll also answer your questions there.

Member

xeno-by replied May 13, 2016

Hello @acdenhartog, thanks for writing! We're all travelling this week, so communication may be delayed. In the coming days, I'll arrange a channel to discuss the SIP and invite everyone to join. I'll also answer your questions there.

@paulp

This comment has been minimized.

Show comment
Hide comment
@paulp

paulp Sep 9, 2016

Contributor

I'll field this one:

That is simply using a new keyword to replace something that an old one already does. Why?

Because the other keyword does a really bad job of it, as outlined here. It was always the wrong keyword, it just took this long for it to be changed.

Contributor

paulp commented Sep 9, 2016

I'll field this one:

That is simply using a new keyword to replace something that an old one already does. Why?

Because the other keyword does a really bad job of it, as outlined here. It was always the wrong keyword, it just took this long for it to be changed.

case q"($name: $_) => $body" =>
body match {
case q"$qual.$_" if qual =:= name =>
q"Ref[${body.tpe}](${name.toString})"

This comment has been minimized.

@SergeantSod

SergeantSod Sep 9, 2016

Why is the example using ${name.toString} in the constructor argument of Ref? As far as I understand, this is the name of the parameter of the lambda, i.e. u. Consequently, the resulting code would be Ref[String]("u").

For the sake of the example I think the desired expansion would be Ref[String]("name"), so the macro would have to capture the name of the selected field in the inner-most case-clause instead.

@SergeantSod

SergeantSod Sep 9, 2016

Why is the example using ${name.toString} in the constructor argument of Ref? As far as I understand, this is the name of the parameter of the lambda, i.e. u. Consequently, the resulting code would be Ref[String]("u").

For the sake of the example I think the desired expansion would be Ref[String]("name"), so the macro would have to capture the name of the selected field in the inner-most case-clause instead.

This comment has been minimized.

@xeno-by

xeno-by Sep 11, 2016

Member

You're correct. I'll fix this soon.

@xeno-by

xeno-by Sep 11, 2016

Member

You're correct. I'll fix this soon.

@odersky

This comment has been minimized.

Show comment
Hide comment
@odersky

odersky Sep 11, 2016

Contributor

If prefix.fTs...(argsN) refers to a partially applied inline method, an error is raised. Eta expansion of inline methods is prohibited.

I think this can be tweaked as follows:

The auto-generated method in an eta expanded inline method is an inline method itself. This means its body will not be inline expanded, but if the eta expansion is subsequently inlined, and the closure is expanded, inlining does happen at that point.

Here's an example of a program that swaps two variables:

object Test {

  inline def swap[T](x: T, x_= : T => Unit, y: T, y_= : T => Unit) = {
    val t = x
    x_=(y)
    y_=(t)
  }

  def main(args: Array[String]) = {
    var x = 1
    var y = 2
    inline def setX(z: Int) = x = z
    inline def setY(z: Int) = y = z
    swap(x, setX, y, setY)
    assert(x == 2 && y == 1)
  }
}

With the tweak I mentioned the body of main expands to (result of -Xprint from dotty compiler):

    var x: Int = 1
    var y: Int = 2
    @inline() def setX(z: Int): Unit = x = z
    @inline() def setY(z: Int): Unit = y = z
    /* inlined from 
      Test.swap[Int](x, 
        {
          @inline() def $anonfun(z: Int): Unit = setX(z)
          closure($anonfun)
        }
      , y, 
        {
          @inline() def $anonfun(z: Int): Unit = setY(z)
          closure($anonfun)
        }
      )
    */ 
      {
        val x: Int = x
        val y: Int = y
        {
          val t: Int = x
          /* inlined from $anonfun(y)*/ 
            {
              /* inlined from setX(y)*/ 
                {
                  x = y
                }
            }
          /* inlined from $anonfun(t)*/ 
            {
              /* inlined from setY(t)*/ 
                {
                  y = t
                }
            }
        }
      }
    assert(x.==(2).&&(y.==(1)))

Note that:

  • the anonymous functions are inline themselves (for the moment, dotty uses @inline as the
    internal way to signal inline, but it also understands the keyword).
  • Because they are inline, their body is not inlined, it's still setX, setY.
  • The final expression inlines perfectly to what we would write as a native swap sequence.
Contributor

odersky commented Sep 11, 2016

If prefix.fTs...(argsN) refers to a partially applied inline method, an error is raised. Eta expansion of inline methods is prohibited.

I think this can be tweaked as follows:

The auto-generated method in an eta expanded inline method is an inline method itself. This means its body will not be inline expanded, but if the eta expansion is subsequently inlined, and the closure is expanded, inlining does happen at that point.

Here's an example of a program that swaps two variables:

object Test {

  inline def swap[T](x: T, x_= : T => Unit, y: T, y_= : T => Unit) = {
    val t = x
    x_=(y)
    y_=(t)
  }

  def main(args: Array[String]) = {
    var x = 1
    var y = 2
    inline def setX(z: Int) = x = z
    inline def setY(z: Int) = y = z
    swap(x, setX, y, setY)
    assert(x == 2 && y == 1)
  }
}

With the tweak I mentioned the body of main expands to (result of -Xprint from dotty compiler):

    var x: Int = 1
    var y: Int = 2
    @inline() def setX(z: Int): Unit = x = z
    @inline() def setY(z: Int): Unit = y = z
    /* inlined from 
      Test.swap[Int](x, 
        {
          @inline() def $anonfun(z: Int): Unit = setX(z)
          closure($anonfun)
        }
      , y, 
        {
          @inline() def $anonfun(z: Int): Unit = setY(z)
          closure($anonfun)
        }
      )
    */ 
      {
        val x: Int = x
        val y: Int = y
        {
          val t: Int = x
          /* inlined from $anonfun(y)*/ 
            {
              /* inlined from setX(y)*/ 
                {
                  x = y
                }
            }
          /* inlined from $anonfun(t)*/ 
            {
              /* inlined from setY(t)*/ 
                {
                  y = t
                }
            }
        }
      }
    assert(x.==(2).&&(y.==(1)))

Note that:

  • the anonymous functions are inline themselves (for the moment, dotty uses @inline as the
    internal way to signal inline, but it also understands the keyword).
  • Because they are inline, their body is not inlined, it's still setX, setY.
  • The final expression inlines perfectly to what we would write as a native swap sequence.
@odersky

This comment has been minimized.

Show comment
Hide comment
@odersky

odersky Sep 12, 2016

Contributor

Inline members also never override other members.

I think this is needlessly restrictive. It would, for instance, prevent us from writing an inline method for Range#foreach.

Can we change the rule to say that functions containing meta blocks are not allowed to override other functions?

Contributor

odersky commented Sep 12, 2016

Inline members also never override other members.

I think this is needlessly restrictive. It would, for instance, prevent us from writing an inline method for Range#foreach.

Can we change the rule to say that functions containing meta blocks are not allowed to override other functions?

@odersky

This comment has been minimized.

Show comment
Hide comment
@odersky

odersky Sep 12, 2016

Contributor

The rules of rewriting mandate the "outside in" style, i.e. calls to inline methods are expanded before possible calls to inline methods in their prefixes and arguments. This is different from how the "inside out" style of def macros, where prefixes and arguments are expanded first. Previously, it was very challenging to take a look at unexpanded trees, but now the metaprogrammer can switch between unexpanded and expanded views using scala.meta.

This is not what is done in the dotty implementation (or what is natural, IMO). Arguments and prefixes are expanded before inline expansion of a call. On the other hand, any expansion
leads to an Inlined node, so one can query what the original call was before the expansion.
Here's the definition of Inlined.

/** A tree representing inlined code.
 *
 *  @param  call      The original call that was inlined
 *  @param  bindings  Bindings for proxies to be used in the inlined code
 *  @param  expansion The inlined tree, minus bindings.
 *
 *  The full inlined code is equivalent to
 *
 *      { bindings; expansion }
 *
 *  The reason to keep `bindings` separate is because they are typed in a
 *  different context: `bindings` represent the arguments to the inlined
 *  call, whereas `expansion` represents the body of the inlined function.
 */
case class Inlined[-T >: Untyped] private[ast] (call: tpd.Tree, bindings: List[MemberDef[T]], expansion: Tree[T])
  extends Tree[T] {
  type ThisTree[-T >: Untyped] = Inlined[T]
}

}

Contributor

odersky commented Sep 12, 2016

The rules of rewriting mandate the "outside in" style, i.e. calls to inline methods are expanded before possible calls to inline methods in their prefixes and arguments. This is different from how the "inside out" style of def macros, where prefixes and arguments are expanded first. Previously, it was very challenging to take a look at unexpanded trees, but now the metaprogrammer can switch between unexpanded and expanded views using scala.meta.

This is not what is done in the dotty implementation (or what is natural, IMO). Arguments and prefixes are expanded before inline expansion of a call. On the other hand, any expansion
leads to an Inlined node, so one can query what the original call was before the expansion.
Here's the definition of Inlined.

/** A tree representing inlined code.
 *
 *  @param  call      The original call that was inlined
 *  @param  bindings  Bindings for proxies to be used in the inlined code
 *  @param  expansion The inlined tree, minus bindings.
 *
 *  The full inlined code is equivalent to
 *
 *      { bindings; expansion }
 *
 *  The reason to keep `bindings` separate is because they are typed in a
 *  different context: `bindings` represent the arguments to the inlined
 *  call, whereas `expansion` represents the body of the inlined function.
 */
case class Inlined[-T >: Untyped] private[ast] (call: tpd.Tree, bindings: List[MemberDef[T]], expansion: Tree[T])
  extends Tree[T] {
  type ThisTree[-T >: Untyped] = Inlined[T]
}

}

@odersky

This comment has been minimized.

Show comment
Hide comment
@odersky

odersky Sep 12, 2016

Contributor

Inline vals and inline parameters. These are guaranteed to be compile-time constants, so an inline value of type T is available inside a meta scope as a regular value of type T.

What about inline call-by-name parameters? Do we allow them? are they also required to be constants?

Contributor

odersky commented Sep 12, 2016

Inline vals and inline parameters. These are guaranteed to be compile-time constants, so an inline value of type T is available inside a meta scope as a regular value of type T.

What about inline call-by-name parameters? Do we allow them? are they also required to be constants?

@odersky

This comment has been minimized.

Show comment
Hide comment
@odersky

odersky Sep 12, 2016

Contributor

By-value and by-name arguments behave differently. By-value arguments are hoisted in temporary variables, while by-name arguments remain as they were. This doesn't affect meta expansions, but it does make a semantic difference for parts of inline definitions that are not inside meta scopes. Note that this and self references always follow the by-value scheme, because there is no syntax in Scala that allows to define them as by-name.

I think this is unfortunate and should be reconsidered. First, it's complicated/confusing to treat cbn and cbv parameters differently here. Second, it can lead to inadvertent code duplication. Third, it is in conflict with the requirement that inline arguments and the inline body should be expanded in different contexts (for instance in what concerns positions). I would prefer if all arguments were hoisted into bindings, vals for cbv parameters and defs for cbn parameters. One should be able to follow through to the actual argument in meta. E.g. we could have a method expansion that expands bound argument vals or defs to their corresponding arguments and that returns other trees unchanged.

Contributor

odersky commented Sep 12, 2016

By-value and by-name arguments behave differently. By-value arguments are hoisted in temporary variables, while by-name arguments remain as they were. This doesn't affect meta expansions, but it does make a semantic difference for parts of inline definitions that are not inside meta scopes. Note that this and self references always follow the by-value scheme, because there is no syntax in Scala that allows to define them as by-name.

I think this is unfortunate and should be reconsidered. First, it's complicated/confusing to treat cbn and cbv parameters differently here. Second, it can lead to inadvertent code duplication. Third, it is in conflict with the requirement that inline arguments and the inline body should be expanded in different contexts (for instance in what concerns positions). I would prefer if all arguments were hoisted into bindings, vals for cbv parameters and defs for cbn parameters. One should be able to follow through to the actual argument in meta. E.g. we could have a method expansion that expands bound argument vals or defs to their corresponding arguments and that returns other trees unchanged.

@MasseGuillaume

This comment has been minimized.

Show comment
Hide comment
@MasseGuillaume
Member

MasseGuillaume commented Sep 19, 2016

(in this sketch, we use an imaginary typeclass `TypeInfo` for that purpose).
`Map` models a restricted subset of SQL SELECT statements via a simple `Node` AST.
```

This comment has been minimized.

@MasseGuillaume

MasseGuillaume Sep 19, 2016

Member

put ```scala to get a syntax highlight

@MasseGuillaume

MasseGuillaume Sep 19, 2016

Member

put ```scala to get a syntax highlight

@xeno-by

This comment has been minimized.

Show comment
Hide comment
@xeno-by

xeno-by Sep 19, 2016

Member

@MasseGuillaume Thanks! Updated the first comment with the link to the rendered document and a todo item.

Member

xeno-by commented Sep 19, 2016

@MasseGuillaume Thanks! Updated the first comment with the link to the rendered document and a todo item.

@jvican

This comment has been minimized.

Show comment
Hide comment
@jvican

jvican Sep 26, 2016

Member

In our September SIP meeting, we decided to split this proposal into two orthogonal parts: inline and meta, to alleviate the difficulty of analyising such a complex proposal. As described in the minutes of our past meeting, these two proposals are not completely independent, and therefore the Committee has compromised to accept them together. @jsuereth and @dragos will give more information in this regard, as well as concrete, actionable feedback to the author of the SIP.

From now on, the following proposal will be tracked with the identifiers SIP-28 and SIP-29.

Member

jvican commented Sep 26, 2016

In our September SIP meeting, we decided to split this proposal into two orthogonal parts: inline and meta, to alleviate the difficulty of analyising such a complex proposal. As described in the minutes of our past meeting, these two proposals are not completely independent, and therefore the Committee has compromised to accept them together. @jsuereth and @dragos will give more information in this regard, as well as concrete, actionable feedback to the author of the SIP.

From now on, the following proposal will be tracked with the identifiers SIP-28 and SIP-29.

@jvican jvican merged commit 1bbee90 into scala:master Sep 26, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Concretely, here's how the transformation works for different references:
**Inline vals and inline parameters**. These are guaranteed to be compile-time constants,
so an inline value of type `T` is available inside a meta scope as a regular value of type `T`.

This comment has been minimized.

@jsuereth

jsuereth Sep 27, 2016

Member

for an inline value that is a function literal, what does this mean?

@jsuereth

jsuereth Sep 27, 2016

Member

for an inline value that is a function literal, what does this mean?

@jsuereth

This comment has been minimized.

Show comment
Hide comment
@jsuereth

jsuereth Sep 27, 2016

Member

Ok, so a few comments here:

  • Would like to see further examination of expanding partially-applied inline methods into synthetic methods (with the caveat that inline parameters must be fully specified).
  • Inline overriding members/methods (Martin's comment) needs to be addressed. It's non-trivial to allow it, and it would dramatically alter the meaning of inline. (i.e. I think a physical implementation would be required at runtime so inheritance would work properly).
  • The hoisting of arguments in generic inline vs. meta expansion still seems 'off' to me. Is it possible that we can describe this away in some other conceptual fashion? i.e. "by default, inline methods that do not have a meta expansion block have the following meta transformation: ...."

I may have a few others, but need to check they're not duplicated already.

Member

jsuereth commented Sep 27, 2016

Ok, so a few comments here:

  • Would like to see further examination of expanding partially-applied inline methods into synthetic methods (with the caveat that inline parameters must be fully specified).
  • Inline overriding members/methods (Martin's comment) needs to be addressed. It's non-trivial to allow it, and it would dramatically alter the meaning of inline. (i.e. I think a physical implementation would be required at runtime so inheritance would work properly).
  • The hoisting of arguments in generic inline vs. meta expansion still seems 'off' to me. Is it possible that we can describe this away in some other conceptual fashion? i.e. "by default, inline methods that do not have a meta expansion block have the following meta transformation: ...."

I may have a few others, but need to check they're not duplicated already.

@dragos

This comment has been minimized.

Show comment
Hide comment
@dragos

dragos Sep 27, 2016

This is a very exciting SIP and brings in a lot of food for thought. I think there’s a good chance this will become the new Scala meta-programming solution, however there’s still work to do. These are my thoughts so far, and more will eventually come as they crystallise in something I can describe properly.

The new mechanism brings in a lighter syntax than existing macros, but overall the improvements seem incremental rather than fundamental: it introduces inlining as an explicit concept, replaces scala.reflect with scala.meta and does away with the separation between macro declaration and macro implementation. However, this simplification comes at the cost of more complex rules for things that are inside meta blocks or not.

The motivating example is an important use-case, but the new approach is compared only to the existing Scala macros. A very similar library has been implemented in Spark SQL (Datasets) without relying on macros so I would like a section motivating the need for compile-time meta-programming. What are the limitations we remove, and how important are they?

Expansion mechanism

Inline method signatures viewed from inside a meta scope change their type to Term. This is a concern since type signatures of inline methods suddenly change. There doesn’t seem to be a precedent in the language for this behaviour. Tool support might also suffer. A natural question then arises: why not write the types as Expr[T], similar to existing macros, and also what LINQ does? I believe this would need to make meta a method modifier, since meta blocks don’t take parameters. It may be that separating inline and meta, but at the same time tweaking inline-meta as a special case is adding complexity. I think this is an area that requires some more thought.

This SIP makes scala.meta the new API for meta-programming. While it certainly lifts the programmer from some compiler implementation details, it brings along its relative immaturity and certain limitations. For example, it collapses the term and type namespace. This means it can’t represent the full Scala language, and I don’t see a compelling argument as to why is that. For example, terms and types with the same name abound (every companion object is a “term”, and its companion class is a type). I’m sure there could be specific work-arounds, but I’d rather see that scala.meta can represent all of Scala. In the coming iterations we will need to make sure the new API is up to the task (and such API is, at least morally, part of this SIP).

dragos commented Sep 27, 2016

This is a very exciting SIP and brings in a lot of food for thought. I think there’s a good chance this will become the new Scala meta-programming solution, however there’s still work to do. These are my thoughts so far, and more will eventually come as they crystallise in something I can describe properly.

The new mechanism brings in a lighter syntax than existing macros, but overall the improvements seem incremental rather than fundamental: it introduces inlining as an explicit concept, replaces scala.reflect with scala.meta and does away with the separation between macro declaration and macro implementation. However, this simplification comes at the cost of more complex rules for things that are inside meta blocks or not.

The motivating example is an important use-case, but the new approach is compared only to the existing Scala macros. A very similar library has been implemented in Spark SQL (Datasets) without relying on macros so I would like a section motivating the need for compile-time meta-programming. What are the limitations we remove, and how important are they?

Expansion mechanism

Inline method signatures viewed from inside a meta scope change their type to Term. This is a concern since type signatures of inline methods suddenly change. There doesn’t seem to be a precedent in the language for this behaviour. Tool support might also suffer. A natural question then arises: why not write the types as Expr[T], similar to existing macros, and also what LINQ does? I believe this would need to make meta a method modifier, since meta blocks don’t take parameters. It may be that separating inline and meta, but at the same time tweaking inline-meta as a special case is adding complexity. I think this is an area that requires some more thought.

This SIP makes scala.meta the new API for meta-programming. While it certainly lifts the programmer from some compiler implementation details, it brings along its relative immaturity and certain limitations. For example, it collapses the term and type namespace. This means it can’t represent the full Scala language, and I don’t see a compelling argument as to why is that. For example, terms and types with the same name abound (every companion object is a “term”, and its companion class is a type). I’m sure there could be specific work-arounds, but I’d rather see that scala.meta can represent all of Scala. In the coming iterations we will need to make sure the new API is up to the task (and such API is, at least morally, part of this SIP).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment