Merge pull request #44 from gwhitney/stratego_eclipse

docs: Update Stratego Tutorial with info about Spoofax/Eclipse

source/langdev/meta/lang/stratego/strategoxt/01-introduction.md

@@ -6,7 +6,7 @@
 
 Program transformation is the mechanical manipulation of a program in order to improve it relative to some cost function `C`, such that `C(P) > C(tr(P))`, i.e., the cost decreases as a result of applying the transformation. The cost of a program can be measured in different dimensions such as performance, memory usage, understandability, flexibility, maintainability, portability, correctness, or satisfaction of requirements. Related to these goals, program transformations are applied in different settings; e.g. compiler optimizations improve performance and refactoring tools aim at improving understandability.
 
-While transformations can be achieved by manual manipulation of programs, in general, the aim of program transformation is to increase programmer productivity by automating programming tasks, thus enabling programming at a higher-level of abstraction, and increasing maintainability and re-usability of programs. Automatic application of program transformations requires their implementation in a programming language. In order to make the implementation of transformations productive, such a programming language should support abstractions for the domain of program transformation.
+While transformations can be achieved by manual manipulation of programs, in general, the aim of program transformation is to increase programmer productivity by automating programming tasks, thus enabling programming at a higher level of abstraction, and increasing maintainability and re-usability of programs. Automatic application of program transformations requires their implementation in a programming language. In order to make the implementation of transformations productive, such a programming language should support abstractions for the domain of program transformation.
 
 Stratego is a language designed for this purpose. It is a language based on the paradigm of rewriting with programmable rewriting strategies and dynamic rules.
 
@@ -15,7 +15,7 @@ Stratego is a language designed for this purpose. It is a language based on the
 
 Term rewriting is an attractive formalism for expressing basic program transformations. A rewrite rule `p1 -> p2` expresses that a program fragment matching the left-hand side pattern `p1` can be replaced by the instantiation of the right-hand side pattern `p2`. For instance, the rewrite rule
 
-    |[ i + j ]| -> |[ k ]| where (i, j) => k
+    |[ i + j ]| -> |[ k ]| where sum(i, j) => k
 
 expresses constant folding for addition, i.e., replacing an addition of two constants by their sum. Similarly, the rule
 
@@ -38,7 +38,7 @@ Stratego solves the problem of control over the application of rules while maint
 
 ## 1.4. Context-Sensitive Transformation
 
-The second problem of rewriting is the context-free nature of rewrite rules. A rule has access only to the term it is transforming. However, transformation problems are often context-sensitive. For example, when inlining a function at a call site, the call is replaced by the body of the function in which the actual parameters have been substituted for the formal parameters. This requires that the formal parameters and the body of the function are known at the call site, but these are only available higher-up in the syntax tree. There are many similar problems in program transformation, including bound variable renaming, typechecking, data flow transformations such as constant propagation, common-subexpression elimination, and dead code elimination. Although the basic transformations in all these applications can be expressed by means of rewrite rules, these require contextual information.
+The second problem of rewriting is the context-free nature of rewrite rules. A rule has access only to the term it is transforming. However, transformation problems are often context-sensitive. For example, when inlining a function at a call site, the call is replaced by the body of the function in which the actual parameters have been substituted for the formal parameters. This requires that the formal parameters and the body of the function are known at the call site, but these are only available higher up in the syntax tree. There are many similar problems in program transformation, including bound variable renaming, typechecking, data flow transformations such as constant propagation, common-subexpression elimination, and dead code elimination. Although the basic transformations in all these applications can be expressed by means of rewrite rules, these require contextual information.
 
 
 

source/langdev/meta/lang/stratego/strategoxt/02-terms.md

@@ -6,7 +6,7 @@
 
 Stratego programs transform terms. When using Stratego for program transformation, terms typically represent the abstract syntax tree of a program. But Stratego does not much care what a term represents. Terms can just as well represent structured documents, software models, or anything else that can be rendered in a structured format.
 
-Generally program text is transformed into a term by means of parsing, and turned back into program text by means of pretty-printing. One way to achieve this is by using [SDF3](../../sdf3/index.md). From now on we will just assume that we have terms that should be transformed and ignore parsing and pretty-printing.
+Generally program text is transformed into a term by means of parsing, and turned back into program text by means of pretty-printing. One way to achieve this is by using [SDF3](../../sdf3/index.md). For most of the examples, we will just assume that we have terms that should be transformed and ignore parsing and pretty-printing. However, when we turn to running examples in the Spoofax environment in the Eclipse IDE, we will rely on SDF3 as that is the primary way to produce terms in Spoofax/Eclipse.
 
 
 ## 2.1. Annotated Term Format
@@ -33,7 +33,7 @@ ATerms are constructed from the following elements:
 * **Tuple**: A tuple `(t1,...,tn)` is a constructor application without a constructor.
 
   Example: `(Var("x"), Type("int"))`
-* **Annotation**: The elements defined above are used to create the structural part of terms. Optionally, a term can be annotated with a list terms. These annotations typically carry additional semantic information about the term. An annotated term has the form `t{t1,...,tn}`.
+* **Annotation**: The elements defined above are used to create the structural part of terms. Optionally, a term can be annotated with a list of terms. These annotations typically carry additional semantic information about the term. An annotated term has the form `t{t1,...,tn}`.
 
   Example: `Lt(Var("n"),Int("1")){Type("bool")}`. The contents of annotations is up to the application.
 
@@ -81,6 +81,8 @@ By pretty-printing the term we get a much more readable term:
       ]
     )
 
+In Spoofax/Eclipse, you will find that in some contexts ATerms are automatically pretty-printed, whereas in others they are simply printed linearly. However, you can obtain assistance with perceiving the structure of any ATerm by writing it into a file with the ".aterm" extension and opening it in the Spoofax Editor in Eclipse. On the right there will be a convenient Outline Navigator which allows you to select any node in the ATerm and see the entire subtree below it highlighted in the editor.
+
 ## 2.4. Signatures
 
 To use terms in Stratego programs, their constructors should be declared in a signature. A signature declares a number of sorts and a number of constructors for these sorts. For each constructor, a signature declares the number and types of its arguments. For example, the following signature declares some typical constructors for constructing abstract syntax trees of expressions in a programming language:
@@ -96,3 +98,5 @@ To use terms in Stratego programs, their constructors should be declared in a si
         Call   : Id  * List(Exp) -> Exp
 
 Currently, the Stratego compiler only checks the arity of constructor applications against the signature. Still, it is considered good style to also declare the types of constructors in a sensible manner for the purpose of documentation.
+
+The situation in Spoofax/Eclipse is even more convenient; if you have an SDF3 language specification, Spoofax will automatically generate a corresponding signature definition that you can import into Stratego.

source/langdev/meta/lang/stratego/strategoxt/03-running-stratego-programs.md

@@ -4,7 +4,7 @@
 
 # 3. Running Stratego Programs
 
-Note: This part refers to deployment in Stratego/XT; update to Spoofax usage
+Note: This entire chapter refers to tools that are a part of Stratego/XT. We will see examples of how to run similar programs in Spoofax/Eclipse toward the end of the next chapter. However, even if you plan to use Spoofax/Eclipse, it's worth at least skimming through this chapter to see Stratego in action and understand how most of the examples throughout this manual will be presented.
 
 Now let's see how we can actually transform terms using Stratego programs. In the rest of this chapter we will first look at the structure of Stratego programs, and how to compile and run them. In the next chapters we will then see how to define transformations.
 

source/langdev/meta/lang/stratego/strategoxt/04-term-rewriting.md

@@ -115,7 +115,88 @@ The first command imports the `prop-eval` module, which recursively loads the ev
 
 The next commands apply the `eval` strategy to various terms.
 
-## 4.2. Adding Rules to a Rewrite System
+## 4.2. Running `prop-eval` in Spoofax/Eclipse
+
+If you'd like to try out some of these Stratego examples in Spoofax/Eclipse, the first step is to define a _concrete syntax_ for Boolean expressions that will parse to the sorts of ATerms that we have been working with. The [SDF3 Manual](../../sdf3/index.md) provides the best introduction to how one might go about doing that, but here is the bulk of an SDF3 syntax definition that will allow us to represent any of the ATerms above:
+
+    context-free syntax
+
+      Prop.True  = <1>
+      Prop.False = <0>
+      Prop.Atom  = String
+      Prop.Not   = <!<Prop>>
+      Prop.And   = <<Prop> & <Prop>>  {left}
+      Prop.Or    = <<Prop> | <Prop>>  {left}
+      Prop.Impl  = [[Prop] -> [Prop]] {right}
+      Prop.Eq    = <<Prop> = <Prop>>  {non-assoc}
+      Prop       = <(<Prop>)> {bracket}
+
+      String = ID
+
+    context-free priorities
+
+      Prop.Not
+      > Prop.And
+      > Prop.Or
+      > Prop.Impl
+      > Prop.Eq
+
+With this grammar in place, the first two examples at the beginning of this chapter (just below the `prop` signature) can be expressed by `(1 -> 0) & 0` and `p & 0`, respectively. So you can see that the concrete syntax will actually make it much easier to construct the example expressions used throughout this manual.
+
+If you'd like to see this in action in Spoofax/Eclipse, you can set up a language with the above grammar. Or you can clone the publicly-available [repository](https://code.studioinfinity.org/glen/spoofax_prop) containing most of the `prop` language examples from this manual.
+
+Either way, you can place either of the above expressions in a file (`syntax/examples/sec4.1_A.spl` or ...`_B.spl` in the repository) and visit it in Eclipse. Then if you select "Syntax > Show parsed AST" from the Spoofax menu, the parsed AST matching our first expressions above will pop up in the editor.
+
+### 4.2.1 Using Editor Services to run a Stratego transformation
+
+Naturally, we'd now like to run `prop-eval` in Spoofax/Eclipse. So we can take the `prop-eval-rules` module above and save it as `trans/prop-eval-rules.str`, with just one small change. Instead of `import prop` in the second line, we can say `import signatures/-`, since Spoofax has written out the signature implied by the grammar for us.
+
+We're also going to have a small module `trans/prop-eval.str` to call the `prop-eval-rules`. It starts out rather similarly to the `prop-eval` for Stratego/XT; here's the first four lines:
+
+    module prop-eval
+    imports libstrategolib prop-eval-rules
+    strategies
+      eval = innermost(E)
+
+Note that we don't have the `main = io-wrap(eval)` line. For Stratego/XT, that was the sort of "glue" we needed to connect the execution environment with the basic `eval` strategy we've defined in Stratego. Similarly, a "glue" expression is needed in Spoofax/Eclipse as well. Because the Eclipse environment is more flexible, the necessary glue is rather more complicated; for now we needn't worry much about its details:
+
+    // Interface eval strategy with editor services and file system
+    do-eval: (selected, _, _, path, project-path) -> (filename, result)
+      with filename := <guarantee-extension(|"eval.aterm")> path
+         ; result   := <eval> selected
+
+How do we now invoke this interface? That's where the Spoofax [Editor Services](../../esv) (ESV) comes in. ESV is responsible, among other things, for the "Spoofax" menu item on the top bar of Eclipse. And you can add a new submenu, which we'll call "Manual", to that menu with a little module `editor/Manual.esv` like this:
+
+    module Manual
+    menus
+      menu: "Manual" (openeditor) (source)
+        action: "prop-eval" = do-eval
+
+Finally, we have to get Spoofax to see our new Stratego and ESV modules. We do this by importing them in the main Stratego and ESV files of the project. In the repository these are in `trans/spoofax_propositional_language.str` and `editor/Main.esv`, respectively. Their beginnings end up looking like:
+
+    module spoofax_propositional_language
+
+    imports
+
+      completion/completion
+      pp
+      outline
+      analysis
+      prop-eval
+    rules // Debugging
+
+and
+
+    module Main
+    imports
+      Syntax
+      Analysis
+      Manual
+    language
+
+Now at last we're ready to invoke the `eval` transformation. Make sure you have your project rebuilt cleanly. Visit a `.spl` file that has the expression you'd like to evaluate, such as `syntax/examples/sec4.1_test1.spl` containing `(1 -> 0 & 1) & 1`. Then select "Spoofax > Manual > prop-eval" from the menu bar to see the value (in this case `False()`. (There's also a ...`_test2.spl` with `(1 -> p & q) & p` for the other example, and you can create your own files for some of the expressions in the Stratego Shell session shown above, if you like.)
+
+## 4.3. Adding Rules to a Rewrite System
 
 Next we extend the rewrite rules above to rewrite a Boolean expression to disjunctive normal form. A Boolean expression is in disjunctive normal form if it conforms to the following signature:
 
@@ -177,3 +258,55 @@ so that we can use it to transform terms:
     Or(And(Not(Atom("r")),Atom("p")),And(And(Atom("p"),Atom("q")),Atom("p")))
 
 [1]: 08-creating-and-analyzing-terms.md "Creating and Analyzing Terms"
+
+## 4.4. Using Spoofax Testing Language to run a Stratego Transformation
+
+We can of course run `prop-dnf` in Spoofax/Eclipse in the same way as before. The `prop-dnf-rules` module is saved verbatim in `trans/prop-dnf-rules.str`, and the `prop-dnf` module becomes the following `trans/prop-dnf.str`:
+
+    module prop-dnf
+    imports libstrategolib prop-dnf-rules
+    strategies
+      dnf = innermost(E)
+
+      // Interface dnf strategy with editor services and file system
+      do-dnf: (selected, _, _, path, project-path) -> (filename, result)
+        with filename := <guarantee-extension(|"dnf.aterm")> path
+           ; result   := <dnf> selected
+
+If you add a "prop-dnf" action to `editor/Manual.esv` calling `do-dnf` and rebuild the project, then you can visit, say, `syntax/examples.sec4.2_test3.spl` containing `(r -> p & q) & p` to produce exactly the DNF shown above.
+
+However, we also want use this example to show another method of running Stratego strategies from the Eclipse IDE.
+
+The [Spoofax Testing Language](../../spt/index.md) (SPT) is a declarative language that provides for a full range of tests for a Spoofax language project. As such, it includes the ability to run an arbitrary Stratego strategy on the results of parsing an arbitrary piece of the language you're working with.
+
+So, we can just take our test3 expression above and make it a part of an SPT test suite, which we will call `test/manual-suite.spt`:
+
+    module manual-suite
+    language Spoofax-Propositional-Language
+
+    test sec4_2_test3 [[
+      (r -> p & q) & p
+    ]] run dnf
+
+Once we have saved this file, the tests run automatically. What does this mean? The file seems to be just "sitting there;" there's no indication that anything is happening. That's because this test we've just written succeeds. All we asked is that Spoofax run the dnf transformation on the results of parsing the test expression. It did that, and the transformation succeeded. So all is well, and no output is generated.
+
+But of course, we want to check the result of the transformation as well. Fortunately, we know what we expect it to be. So we can change the test like so:
+
+    test sec4_2_test3_ex [[
+      (r -> p & q) & p
+    ]] run dnf to Or(And(Not(Atom("r")),Atom("p")),
+                     And(And(Atom("p"),Atom("q")),Atom("p")))
+
+Now if there is no error or warning on this test then you know the `dnf` strategy produced the result shown in the `to` clause, and otherwise, the actual result will be shown in the error popup.
+
+What if you _don't_ know what the expression is going to produce? Then you can just put a dummy expression like `Atom("x")` in the `to` clause, and you will be sure to get an error. The error popup will show the actual transformation results. But beware! The results will be hard to read because of the annotations that Spoofax adds to track where in the source code each part of the AST originates. (For example, in the example above we get
+
+    Got: Or(And(Not(Atom("r"{TermIndex("test/manual-suite.spt",1)}){TermIndex
+     ("test/manual-suite.spt",2)}),Atom("p"{TermIndex("test/manual-suite.spt",9)})
+     {TermIndex("test/manual-suite.spt",10)}),And(And(Atom("p"{TermIndex("test/manual-
+     suite.spt",3)}){TermIndex("test/manual-suite.spt",4)},Atom("q"{TermIndex
+     ("test/manual-suite.spt",5)}){TermIndex("test/manual-suite.spt",6)}){TermIndex
+     ("test/manual-suite.spt",7)},Atom("p"{TermIndex("test/manual-suite.spt",9)})
+     {TermIndex("test/manual-suite.spt",10)}))
+
+Nevertheless, with the editor outliner you can puzzle out what your transformation has done. The fact remains that it is most practical to put the actual expected result of the transformation in the `to` clause.