Import development documentation, merge with IntelliJ documentation

index.rst

@@ -47,21 +47,21 @@ Table of Contents
 
 .. toctree::
    :maxdepth: 3
-   :caption: Spoofax Development
+   :caption: Development
 
+   Getting Started <source/dev/start>
+   Maven <source/dev/maven>
    Build <source/dev/build>
-   Bootstrap <source/dev/bootstrap>
-   Release <source/dev/release>
+   Develop <source/dev/dev>
    Contribute <source/dev/contribute>
-
+   Release <source/dev/release>
+   Internals <source/dev/internals/index>
+
 .. toctree::
    :maxdepth: 3
    :caption: IntelliJ IDEA Plugin Development
 
-   Building <source/intellijdev/building>
-   Releasing <source/intellijdev/releasing>
    Contributing <source/intellijdev/contributing>
-   Architecture <source/intellijdev/architecture>
 
 .. toctree::
    :maxdepth: 3

source/dev/build.md

@@ -1,5 +1,131 @@
 # Building Spoofax
 
+This section describes how to build Spoofax from scratch, on the command-line.
+
+## Supported platforms
+
+OSX, Linux, and Windows are supported.
+
+## Requirements
+
+1. **Git 1.8.2 or higher** is required to check out the source code from our GitHub repositories. Instructions on how to install Git for your platform can be found here: <http://git-scm.com/downloads>. If you run OSX and have [Homebrew](http://brew.sh/) installed, you can install Git by executing `brew install git`. Confirm your Git installation by executing `git version`.
+2. **Maven 3.3.9 or higher** is required to build Spoofax and StrategoXT, refer to the [Setting up Maven](maven.md) section for instructions on how to install Maven.
+3. **MetaBorg Maven artifacts** are required since the build depends on artifacts from previous builds for bootstrapping purposes. Follow [Using MetaBorg Maven artifacts](maven.md) to make these artifacts available to Maven.
+4. **Python 3.4 or higher** is used to orchestrate the build. Instructions on how to install Python for your platform can be found here: <https://www.python.org/downloads/>. If you run OSX and have [Homebrew](http://brew.sh/) installed, you can install Python by executing `brew install python3`. Confirm your Python installation by executing `python3 --version`.
+5. **Pip 7.0.3 or higher** is used to download some python dependencies. A version comes preinstalled with Python 3, but you need to make sure that you have version 7.0.3 or higher. To upgrade to the newest version use `pip install --upgrade pip` or `pip3 install --upgrade pip`, depending on if your OS uses a different pip for Python 3. Confirm using `pip --version` or `pip3 --version`.
+
+## Cloning the source code
+
+Clone the source code from the [spoofax-releng](https://github.com/metaborg/spoofax-releng) repository with the following commands:
+
+```bash
+git clone https://github.com/metaborg/spoofax-releng.git
+cd spoofax-releng
+git submodule update --init --remote --recursive
+```
+
+Cloning and updating submodules can take a while, since we have many submodules and some have a large history.
+
+## Building
+
+To build Spoofax, simply execute:
+
+```bash
+./b build all
+```
+
+This downloads the latest StrategoXT, and builds Spoofax. If you also want to build StrategoXT from scratch, execute:
+
+```bash
+./b build -st all
+```
+
+The `-s` flag build StrategoXT instead of downloading it, and `-t` skips the StrategoXT tests since they are very lengthy.
+The `all` part of the command indicates that we want to build all components. If you would only like to build the Java components of Spoofax, and skip the Eclipse plugins, execute:
+
+```bash
+./b build java
+```
+
+Use `./b build` to get a list of components available for building, and `./b build --help` for help on all the command-line flags and switches.
+
+```eval_rst
+.. note:: If you have opened a project in the repository in Eclipse, you **must turn off** :menuselection:`Project --> Build Automatically` in Eclipse, otherwise the Maven and Eclipse compilers will interfere and possibly fail the build. After the Maven build is finished, enable :guilabel:`Build Automatically` again.
+```
+
+## Updating the source code
+
+If you want to update the repository and submodules, execute:
+
+```bash
+git pull --rebase
+./b checkout
+./b update
+```
+
+The `git pull` command will update any changes in the main repository. The `./b checkout` command will check out the correct branches in all submodules, because Git does not do this automatically. The `./b update` command will update all submodules.
+
+## Switching to a different branch
+
+Switching to a different branch, for example the `spoofax-release` branch, is done with the following commands:
+
+```bash
+git checkout spoofax-release
+git pull --rebase
+git submodule update --init --remote --recursive
+./b checkout
+./b update
+```
+
+## Troubleshooting
+
+### Resetting and cleaning
+
+If updating or checking out a branch of submodule fails (because of unstaged or conflicting changes), you can try to resolve it yourself, or you can reset and clean everything. Reset and clean all submodules using:
+
+```bash
+./b reset
+./b clean
+```
+
+```eval_rst
+.. warning:: Resetting and cleaning DELETES UNCOMMITTED AND UNPUSHED CHANGES, which can cause PERMANENT DATA LOSS. Make sure all your changes are committed and pushed!
+```
+
+### Weird compilation errors
+
+If you get any weird compilation errors during the build, make sure that <span class='menuselection'>Project ‣ Build Automatically</span> is turned off in Eclipse.
+
+## IntelliJ plugin
+
+The IntelliJ plugin is currently built separately from the spoofax-releng repository and build scripts.
+To build the IntelliJ IDEA Spoofax plugin locally, you first need to get the source [from GitHub][1].
+Choose one of the branches:
+
+* [`master` branch][2]: stable build of current release. ([ZIP][4])
+* [`develop` branch][3]: unstable build for next release. ([ZIP][5])
+
+Either clone the repository locally or download and extract the ZIP archive.
+
+Next you need to invoke Gradle to build the project.
+Simply execute the following command from the main project's root folder (i.e. <span class='file'>org.metaborg.intellij/</span>):
+
+```bash
+./gradlew clean build
+```
+(On Windows `gradlew clean build`.)
+
+Alternatively, you can import the project in IntelliJ and build it from there.
+
+
+[1]: https://github.com/metaborg/spoofax-intellij
+[2]: https://github.com/metaborg/spoofax-intellij/tree/master
+[3]: https://github.com/metaborg/spoofax-intellij/tree/develop
+[4]: https://github.com/metaborg/spoofax-intellij/archive/master.zip
+[5]: https://github.com/metaborg/spoofax-intellij/archive/develop.zip
+
+## Bootstrapping
+
 ```eval_rst
 .. todo:: This part of the documentation has not been written yet.
 ```

source/dev/dev.md

@@ -0,0 +1,107 @@
+# Developing Spoofax
+
+If you are developing a project that is included in Spoofax it is recommended to set up a development environment.
+This section describes how to set up such a development environment.
+
+## Requirements
+
+1. A working **Spoofax build** is required before being able to develop. Follow the [Building Spoofax](build.md) guide for instructions on how to build Spoofax.
+
+## Eclipse
+
+Currently, an Eclipse development environment is the most supported environment.
+
+### Generating an Eclipse instance
+
+The `b` script in the <span class='file'>spoofax-releng</span> repository can generate an Eclipse installation for you.
+Change directory into the <span class='file'>spoofax-releng</span> repository and run:
+
+```bash
+./b gen-dev-spoofax -d ~/eclipse/spoofax-dev
+```
+
+This will download and install Eclipse into <span class='file'>~/eclipse/spoofax-dev</span> with the right plugins and <span class='file'>eclipse.ini</span> for Spoofax development. The latest nightly version of the Spoofax plugin will be installed into that Eclipse. If you would like to install your locally built Spoofax plugin instead, pass the `-l` flag:
+
+```bash
+./b gen-dev-spoofax -l -d ~/eclipse/spoofax-dev
+```
+
+Generating an Eclipse installation can take several minutes. After it's done generating, open the Eclipse installation and confirm that it works by creating a Spoofax entity project.
+
+### Fixing Eclipse settings
+
+Some Eclipse settings unfortunately have sub-optimal defaults. Go to the Eclipse preferences and set these options:
+
+* <span class='menuselection'>General</span>
+	* Enable: <span class='guilabel'>Keep next/previous editor, view and perspectives dialog open</span>
+* <span class='menuselection'>General ‣ Startup and Shutdown</span>
+	* Enable: <span class='guilabel'>Refresh workspace on startup</span>
+* <span class='menuselection'>General ‣ Workspace</span>
+	* Enable: <span class='guilabel'>Refresh using native hooks or polling</span>
+* <span class='menuselection'>Maven</span>
+	* Enable: <span class='guilabel'>Do not automatically update dependencies from remote repositories</span>
+	* Enable: <span class='guilabel'>Download Artifact Sources</span>
+	* Enable: <span class='guilabel'>Download Artifact JavaDoc</span>
+* <span class='menuselection'>Maven ‣ User Interface</span>
+  * Enable: <span class='guilabel'>Open XML page in the POM editor by default</span>
+* <span class='menuselection'>Run/Debug ‣ Launching</span>
+	* Disable: <span class='guilabel'>Build (if required) before launching</span>
+
+### Developing
+
+Import the projects you'd like to develop.
+To import Java and language projects, use <span class='menuselection'>Import ‣ Maven ‣ Existing Maven Projects</span>.
+Eclipse plugins are still imported with <span class='menuselection'>Import ‣ General ‣ Existing Projects into Workspace</span>.
+
+### Running
+
+To test your changes in the Spoofax Eclipse plugin, import the `org.metaborg.spoofax.eclipse` project from the `spoofax-eclipse` repository, which provides launch configurations for starting new Eclipse instances. Press the little down arrow next to the bug icon (next to the play icon) and choose `Spoofax with core (all plug-ins)` to start a new Eclipse instance that contains your changes.
+
+Some gotcha's:
+
+* When starting a new Eclipse instance using `Spoofax with core (all plug-ins)`, Eclipse might report problems about `org.eclipse.jdt.annotation`, `org.metaborg.meta.lang.spt.testrunner.cmd`, and `org.metaborg.meta.lang.spt.testrunner.core`. These problems can be ignored.
+* If you change a language and want to test it in a new Eclipse instance, import that language's corresponding Eclipse plugin project. For example, `org.metaborg.meta.lang.nabl` has Eclipse plugin project `org.metaborg.meta.lang.nabl.eclipse`. Then compile both those projects from the command-line (don't forget to turn off build automatically in Eclipse), and start a new Eclipse instance.
+
+### Troubleshooting
+
+If there are many errors in a project, try updating the Maven project.
+Right click the project and choose <span class='menuselection'>Maven -> Update Project...</span>, uncheck <span class='guilabel'>Clean projects</span> in the new dialog and press OK.
+This will update the project from the POM file, update any dependencies, and trigger a build.
+If this does not solve the problems, try it again but this time with <span class='guilabel'>Clean projects</span> checked.
+Note that if you clean a language project, it has to be rebuilt from the command-line. Restarting Eclipse and repeating these steps may also help.
+
+Multiple projects can be updated by selecting multiple projects in the package/project explorer, or by checking projects in the update dialog.
+
+### Advanced: developing from scratch
+
+In some cases it can be beneficial to have full control over all projects, instead of relying on Maven artifacts and the installed Spoofax plugin.
+Only follow this approach if you know what you are doing!
+To develop from scratch, uninstall Spoofax from Eclipse, and import all projects from `spoofax-releng` into the workspace.
+
+If you change a language project, build them on the command-line, because languages cannot be built inside Eclipse without the Spoofax plugin.
+
+## IntelliJ
+
+To run the built plugin inside a special sandbox-instance of IntelliJ IDEA, execute the following command:
+
+```
+./gradlew runIdea
+```
+
+Alternatively, in IntelliJ IDEA you can invoke the _IntelliJ Plugin_ run/debug configuration.
+You can use this to run or debug the IntelliJ IDEA plugin code.
+However, this cannot be used to debug the JPS Spoofax build process.
+
+To debug the JPS Spoofax build process, you need to execute the following command:
+
+```
+./gradlew debugJps
+```
+
+or invoke the _IntelliJ Plugin (Debug JPS)_ run configuration (_not debug_) from IntelliJ.
+
+Then from the sandbox IntelliJ IDEA instance you start a build.
+IntelliJ will wait for a debugger to be attached to port 5005.
+Attach a debugger, and the build will continue.
+From the Spoofax plugin's IntelliJ IDEA project, you can invoke the _JPS Plugin_ debug configuration to attach the
+debugger.

source/dev/internals/index.rst

@@ -0,0 +1,8 @@
+=====================
+Development Internals
+=====================
+
+.. todo:: This part of the documentation has not been written yet.
+
+.. toctree::
+   :maxdepth: 1