Add differences in project structure. Add simple lexer for Stratego a…

…nd ESV

_static/custom.css

@@ -1,9 +1,3 @@
-/*code.docutils.literal {
-	color: #222;
-	background: #eee;
-	font-family: monospace;
-}*/
-
 .menuselection, .guilabel, .file, kbd {
 	background: #eee;
 }

conf.py

@@ -370,7 +370,54 @@
 # If false, no index is generated.
 #epub_use_index = True
 
+# -- Options for inline highlighting ----------------------------------------------
+
+inline_highlight_literals = False
+
+# -- Setup ------------------------------------------------------------------------
+
+import re
+from pygments.lexer import RegexLexer, words
+from pygments.token import *
+
+class StrategoLexer(RegexLexer):
+  name = 'Stratego'
+  aliases = ['stratego', 'str']
+  filenames = ['*.str']
+
+  tokens = {
+      'root': [
+        (words(('strategies','rules','where','with'), suffix=r'\b'), Keyword),
+        (r'(\+|\?|!)', Operator),
+        (r'"[^"^\n]*"', Literal.String),
+        (r'\d+', Literal.Number),
+        (r'[\w_-]+', Name.Variable),
+        (r'[\,\|\[\]\(\)\{\}\<\>\;\:\=]', Text.Punctuation),
+        (r'\s+', Text.Whitespace),
+        (r'.', Text),
+      ]
+  }
+
+class ESVLexer(RegexLexer):
+  name = 'ESV'
+  aliases = ['esv']
+  filenames = ['*.esv']
+
+  tokens = {
+      'root': [
+          (words(('language','line comment','block comment','fences'), suffix=r'\b'), Keyword),
+          (r'"[^"^\n]*"', Literal.String),
+          (r'[\.\,\|\[\]\(\)\{\}\<\>\;\:\*]', Text.Punctuation),
+          (r'\s+', Text.Whitespace),
+          (r'.', Text),
+      ]
+  }
+
+# -- Setup ------------------------------------------------------------------------
+
 def setup(app):
   app.add_config_value('recommonmark_config', {}, True)
   app.add_transform(AutoStructify)
   app.add_stylesheet("custom.css")
+  app.add_lexer("str", StrategoLexer())
+  app.add_lexer("esv", ESVLexer())

requirements.txt

@@ -2,3 +2,5 @@ sphinx>=1.3.6
 sphinx-autobuild>=0.6.0
 recommonmark>=0.4.0
 sphinx_rtd_theme>=0.1.9
+sphinxcontrib-inlinesyntaxhighlight>=0.2
+pygments>=2.1.3

source/release/migrate/2.0.0.md

@@ -30,17 +30,27 @@ A *language* is the more abstract notion of a language, which has multiple langu
 
 An *end-user project* is a project with programs of an end-user language, in contrast to a language specification project which has programs of meta-languages. For example, a Java project is a Java end-user project, whereas the JDK project is a language specification project.
 
-### General
+### Project structure
 
-project structure
+The project structure of language specification projects has significantly changed. The biggest change is that these projects are not longer Eclipse (plugin) projects, so that they can be used outside of the Eclipse platform as well. Ant build files have also been removed since we do not use Ant to build projects any more. Many ESV files have been deprecated, and all generated ESV files in the <span class='file'>editor</span> directory have been removed.
 
-generated files
+The following files and directories are no longer part of the project structure:
 
-### Eclipse
+* Ant build: <span class='file'>.externalToolBuilders</span>, <span class='file'>build.generated.xml</span>, <span class='file'>build.main.xml</span>
+* Eclipse plugin: <span class='file'>plugin.xml</span>, <span class='file'>META-INF</span>
+* Eclipse project: <span class='file'>.settings</span>, <span class='file'>.classpath</span>, <span class='file'>.project</span>, <span class='file'>build.properties</span>
+* Refactoring: <span class='file'>lib/refactor-common.generated.str</span>
+* Deprecated ESV files: <span class='file'>editor/langname-Completions.esv</span>, <span class='file'>editor/langname-Folding.esv</span>, <span class='file'>editor/langname-Outliner.str</span>, <span class='file'>editor/langname-Refactorings.esv</span>
+* Generated ESV files: <span class='file'>editor/langname-\*.generated.esv</span>, <span class='file'>editor/langname-Outliner.generated.str</span>
+
+The following generated files and directories still exist, but do not have to be committed to source control any more:
+
+* <span class='file'>lib/editor-common.generated.str</span>
+* <span class='file'>src-gen</span>
 
-#### Projects
+When importing a language specification project as
 
-Language specification projects are no longer Eclipse plugins, so that they can be used outside of the Eclipse platform as well.
+### Eclipse
 
 ##### Importing
 
@@ -144,7 +154,7 @@ If you’re using a NaBL/TS based analysis, perform the following changes:
 * NaBL files are now generated into <span class='file'>src-gen/names</span>, fix imports to NaBL files, delete old generated NaBL files.
 * TS files are now generated into <span class='file'>src-gen/types</span>, fix imports to TS files, delete old generated TS files.
 * The `editor-analyze` calls have been changed. Remove `analysis-single-default-interface`, `analysis-multiple-default-interface`, and `editor-analyze`. Replace it with:
-```
+```str
 editor-analyze = analyze-all(pre-analysis, post-analysis, pp-message|<language>)
 ```
 with the `pre-analysis`, `post-analysis`, and `pp-message` arguments that you were using before.
@@ -156,16 +166,21 @@ with the `pre-analysis`, `post-analysis`, and `pp-message` arguments that you we
 
 #### ESV
 
+* The following ESV files are now deprecated, delete and remove any imports to these files:
+  * <span class='file'>editor/langname-Colorer.esv</span>
+  * <span class='file'>editor/langname-Folding.esv</span>
+  * <span class='file'>editor/langname-Refactorings.esv</span>
 * Previously generated ESV files in the <span class='file'>editor</span> directory are not generated any more. Delete the generated files and remove the imports to generated files.
-* The colorer ESV file is generated to <span class='file'>src-gen/editor/Colorer.generated.esv</span>, import it with `imports editor/Colorer.generated` in ESV.
-* The Syntax ESV file is no longer generated. If you were using the defaults from the generated file, add them to your language’s Syntax ESV file:
-```
+* The colorer ESV file is now generated to <span class='file'>src-gen/editor/Colorer.generated.esv</span>, import it with `imports editor/Colorer.generated` in an ESV file.
+* The generated syntax ESV file is no longer generated. If you were using the defaults from the generated file, add them to an ESV file:
+```esv
 language
 
   line comment  : "//"
   block comment : "/*" * "*/"
   fences        : [ ] ( ) { }
 ```
+* The outliner (<span class='file'>editor/langname-Outliner.str</span>) must be moved to the <span class='file'>trans</span> directory. The <span class='file'>editor</span> directory is no longer a source directory for Stratego.
 
 #### Ant build customization
 

source/release/note/2.0.0.md

@@ -47,6 +47,8 @@ A few features didn't make it to Spoofax 2.0, with the biggest one being complet
 
 Both syntactic and semantic completions were already very dodgy in Spoofax 1.5, only working in some specific cases. This is why we did not port the completion algorithm from Spoofax 1.5 to 2.0, and are instead working on a new completion algorithm that will be included in a future version.
 
+Refactorings were already broken in Spoofax 1.5, so we did not port refactorings to Spoofax 2.0. In the future we will revisit refactorings for Spoofax 2.0 with our new meta-languages.
+
 The Spoofax modelware component was not ported to Spoofax 2.0 since we do not have the knowledge to port this component.
 
 Folding, realtime builders, and the <span class='file'>eclipse.ini</span> check are minor features that are not implemented in 2.0, but may be implemented in the future.