DirectEmbedding

An experimental macro-based library for Embedded DSLs.

DirectEmbedding for DSLs

DirectEmbedding is an experimental project that attempts to provide direct embedding in Scala for Domain Specific Languages (DSLs). Although, this project is currently exploratory, this library could become a very useful tool. For instance, it could replace the complex shadow embedding of Yin-Yang library in existing or future projects such as Slick.

This library provides one effortless logic for the reification of embedded DSLs in Scala. It does not require the DSL author to:

• have knowledge of the Reflection API
• take care of overloading resolution by himself
• write code relying on the types and the count of the arguments
• write verbose code
• duplicate code

Our solution makes use of annotations and macros to achieve this goal.

##Table of contents:

1. Overview of the concept
2. Details
3. Project Structure
4. Usage
5. References
6. Licence

--

Overview

Writing embedding DSLs implies to go through the difficult task of reification. Reification is a conversion of domain-specific and Scala operations to an intermediate representation (IR). For example, the function take(x: Int): Query[T] is written in Scala it will be converted to a corresponding DSL IR object Take. The naive method would code conditional statements in order to identify which function has been called. Identifying is strenuous and redundant because it must check many criteria, take in account aspects such as overriding of the function and this for all functions or classes. It is not a best practices and it creates as much logics of reification than there is cases to reify. Hopefully, this can be facilitated by libraries and DirectEmbedding is one of them.

More details in Quasiquotes

Attach Metadata to declarations

The idea developed by DirectEmbedding is to attach metadata about the corresponding IR to functions and classes declarations. Thus, it gets rid of the demanding task of identification. It becomes straightforward to solve the previous issues. When it was required to verify the name of the function, its types, its types arguments, its arguments count, now this is declared along the function. The overriding is also very simple. The reification will apply to the attached IR of the overridden so there is no need to verify anything like count of arguments or their types. Because the attached IR must match the overridden function, otherwise the compilation fails.

Annotate

Scala annotations can attribute an object to declarations. Thus, annotations permit to attach metadata, that is to say, to the IR. The annotation is accessible through the symbol of the AST.

Reify

DirectEmbedding reifies the DSL during the compilation. This means that the given AST for an annotated function will be reified into the DSL representation at compile-time. In order to do so, there is the need to extract the information of all the possible ASTs that can be generated by an annotated declaration. Once the arguments, the type arguments and the symbol of the annotated code are extracted from the AST then the reification occurs via a macro. The macro applies the arguments and types arguments to the obtained IR from the symbol. The macro returns the reified tree i.e. the IR with the arguments and the type arguments correctly applied.

With DirectEmbedding, there is not special difficulty for the identification, it does the hard task of extracting the information from the different ASTs and using the macros, the solution is elegant and proposes only one reification logic. Pretty simple!

An example

On the side of the DSL's developper

• Let's imagine, we want to define for a DSL a function take() that returns a Query[T]:

   	class Query[T] {
   		val take(x: Int): Query[T] = ???
   	}

• For this DSL, the function take() should be reified into the following IR:

   	case class Take[T](self: Exp[QueryIR[T]], n: Exp[Int])
   	   extends Exp[Query[T]]

• Thus, the function take() needs to be annotated:

   	class Query[T] {
   		@reifyAs(Take) //Annotation with its corresponding IR
   		val take(x: Int): Query[T] = ???
   	}

On the DSL's user side

• One user calls the function take() in his code:

      lift {
         	new Query[Movie].take(3)
      }

• The function is reified inside at compile-time into:

      Expr[T](Take.apply[Int](QueryIR.apply[Movie], 3))

What happened?

• A Scala AST is generated and caught
• The annotation, the arguments and the type arguments are extracted
• A macro reified the AST into the result

Details

Quasiquotes?

	case q"$a.take($b)" if a.tpe =:= typeOf[Query[_]]
  	 		&& b.tpe =:= typeOf[Int] => Take(a, b)

The code above shows what would be reification without DirectEmbedding. Although, in this case, we use quasiquotes, there is none of the advantages of DirectEmbedding:

1. no knowledge of the Reflection API
   • this code obviously necessitates knowledge of the Reflection API

• no overloading resolution

  • if take(x: Int, b: Boolean): Query[T] overrides take(x: Int): Query[T] then another conditional statement would be needed.

   	case q"$a.take($b, $c)" if a.tpe =:= typeOf[Query[_]]
    		&& b.tpe =:= typeOf[Int]
    		&& c.tpe =:= typeOf[Boolean] => Take(a, b, c)

• no dependence with types and count of arguments

  • as shown in the previous example, a new argument implies a new code

• no verbosity

  • this kind of code can become illegible

• no duplication of code

  • again in the override example, there is unnecessary code duplication

Project Structure

Component	Description
directembedding/... /DirectEmbedding.scala	DirectEmbedding code: reification code with macro
dsls/main/.../BasicSpec.scala	Test: Intermediate Representation
dsls/test/.../TestBase.scala	Test: Corners cases tests

DirectEmbedding.scala

This file contains the reification code. It consists of the definition of:

• reifyAs()

      class reifyAs(to: Any) extends scala.annotation.StaticAnnotation

  • This defines the annotation reifyAs which accepts Any object so the IR can be attached into the symbols

• and lift()

   	def lift[T](c: Context)(block: c.Expr[T]): c.Expr[T] = {
  			import c.universe._
   	    class LiftingTransformer extends Transformer {
   	    	...
   	    }
   	}

  • This is the method that encompasses all the reification process.

The class LiftingTransformer defines:

• reify()

   	def reify(methodSym: Symbol, targs: List[Tree],
   		args: List[Tree]): Tree = { ... }

  • This code uses the macro with its parameters to reify the captured function into its IR.

• and transform()

   	override def transform(tree: Tree): Tree = { ... }

  • transform() pattern matches over the different ASTs to extract the essential data for reify() that is to say the symbol, the arguments and the type arguments.

TestBase.scala

This file contains the functions and IR that represent a DSL. It is used to for testing purpose in BasicSpec.scala

Example for a class:

case object QueryIR[T] extends Exp[Query[T]] // Note: IR extends the real returned type

@reifyAs(QueryIR)
class Query[T] {
	...
}

Example for a function with many arguments:

case class TakeList[T](self: Exp[QueryIR[T]], x: Exp[Int]*) extends Exp[T] // Note: all function have as first argument a self

@reifyAs(QueryIR)
class Query[T] {
	  @reifyAs(TakeList)
	  def take[T](x: Int*): List[Query[T]] = ???
}

BasicSpec.scala

This fils contains the tests.

Below the test for take():

"lift" should "work with Query methods with take" in {
  testReify(implicit collec =>
    lift {
      new Query[Int].take(3)
    }) should be(List(Take[Int](QueryIR[Int](), 3)))
}

Usage

The project has been tested under Sbt 0.13.6 and Scala 2.11.2

Dependencies

The project depends on ScalaTest 2.2.1 library, copy paste the hereafter dependency into the build.sbt in <sbtRootFolder>/0.13/plugins/

libraryDependencies += "org.scalatest" % "scalatest_2.11" % "2.2.1" % "test"

Launching the project

SBT is required to compile the project.

To launch the tests:

 sbt test

To configure the project for Eclipse run the command:

 sbt eclipse

Progress

Todos	Done	Details
values	Yes	val x = ...
function	Yes	def foo: ...
function with args	Yes	def foo(x: Int): Int = ...
function with targs	Yes	def foo[T, U]: (T, U) = ...
function with args and targs	Yes	def foo[T, U](t: T, u: U): (T, U) = ...
objects	Yes
nested objects	Yes
classes	Yes
language specification	No	if, while, read-var
operator	No
recursion	No
override	No

References

The following links are interesting papers concerning the context of this project:

• Experimental direct embedding for Slick
• An Embedded Query Language in Scala
• Yin-Yang: Concealing the Deep Embedding of DSLs

License

DirectEmbedding is licensed under the EPFL License.