Merge branch 'master' of github.com:evanphx/rubinius

web/doc/en/bytecode-compiler.markdown

@@ -28,9 +28,9 @@ stages, as well as their inputs and outputs, are illustrated below.
 1. [Parser Stage](/doc/en/bytecode-compiler/parser/)
 1. [AST](/doc/en/bytecode-compiler/ast/)
 1. [Generator Stage](/doc/en/bytecode-compiler/generator/)
-1. Encoder Stage
-1. Packager Stage
-1. Writer Stage
+1. [Encoder Stage](/doc/en/bytecode-compiler/encoder/)
+1. [Packager Stage](/doc/en/bytecode-compiler/packager/)
+1. [Writer Stage](/doc/en/bytecode-compiler/writer/)
 1. Printers
 1. [Transformations](/doc/en/bytecode-compiler/transformations/)
 1. [Customizing the Pipeline](/doc/en/bytecode-compiler/customization/)

web/doc/en/bytecode-compiler/encoder.markdown

@@ -0,0 +1,39 @@
+---
+layout: doc_en
+title: Encoder Stage
+previous: Generator Stage
+previous_url: bytecode-compiler/generator
+next: Packager Stage
+next_url: bytecode-compiler/packager
+---
+
+Once the generator has processed the AST, it needs to properly encode
+the bytecode. This stage is very simple, and is mostly for
+recordkeeping.
+
+The Encoder stage is responsible for two things:
+
+* converting the stream of instructions created in the previous stage
+  (when the AST nodes called `bytecode`) into an instruction sequence.
+* validating that the stack does not underflow, overflow, or otherwise
+  contain faulty semantics.
+
+It performs these steps on the main Generator object, as well as any
+child Generators (generators for blocks, methods or other structures
+encountered inside of the main body).
+
+Once this stage is complete, it passes the encoded Generator object to
+the next stage, the Packager stage.
+
+## Files Referenced
+
+* *lib/compiler/generator.rb*: The `encode` method in the file performs
+  the bulk of the work in this stage.
+
+## Customization
+
+Since this stage is so simple, you will not need to customize it. You
+might want to instrument it (for instance, for profiling or printing).
+To learn more about general-purpose customization, which you can use to
+instrument any stage, check out [Customizing the Compiler
+Pipeline](/bytecode-compiler/customization/).

web/doc/en/bytecode-compiler/packager.markdown

@@ -0,0 +1,69 @@
+---
+layout: doc_en
+title: Packager Stage
+previous: Encoder Stage
+previous_url: bytecode-compiler/encoder
+next: Writer Stage
+next_url: bytecode-compiler/writer
+---
+
+Once the Generator has been properly encoded in the Encoder stage,
+Rubinius packages the bytecode up as a CompiledMethod by creating a new
+CompiledMethod and setting a number of properties on it.
+
+These properties are available on any CompiledMethod. You can retrieve
+the CompiledMethod from a Ruby Method object by calling `executable` on
+it.
+
+* *iseq*: a Tuple containing the raw instruction sequence
+* *literals*: a Tuple contaning the literals used in the method.
+  Literals are used internally by Rubinius for values like Strings, and
+  are used by the `push_literal` and `set_literal` instructions.
+* *lines*: an Array containing the first instruction pointer for each
+  line represented by the bytecode
+* *required_args*: the number of arguments required by the method
+* *total_args*: the total number of arguments, including optional
+  arguments but not `*args`
+* *splat*: the position of the splat argument, if any
+* *local_count*: the number of local variables, including parameters
+* *local_names*: a Tuple containing a list of all of the local variable
+  names. The first names will be the required, optional, splat and block
+  arguments, in that order.
+* *file*: the name of the file that will be used in stack traces and
+  other debugging information
+* *name*: the name of the method
+* *primitive*: the name of the primitive associated with this method, if
+  any
+* metadata: it is possible to store additional structured metadata on a
+  compiled method. The compiled method has a piece of metadata named
+  `for_block` with the value `true` if the original generator was
+  created for a block.
+
+The Packager stage also makes sure that any child generators (such as
+generators for blocks or methods) are also converted into compiled
+methods. These child compiled methods are included in the literals tuple
+of the parent compiled method.
+
+Once the Generator has finished packaging itself as a CompiledMethod, it
+invokes the Writer stage, passing in the CompiledMethod as its input.
+
+## Files Referenced
+
+* *kernel/bootstrap/compiled_method.rb*: the basic implementation of
+  CompiledMethod, mostly composed of wiring up primitives
+* *kernel/common/compiled_method.rb*: a more robust implementation of
+  CompiledMethod, a combination of primitive methods and methods written
+  in Ruby
+* *vm/builtin/compiledmethod.cpp*: the C++ implementation of the
+  CompiledMethod primitives
+* *lib/compiler/generator.rb*: The implementation of the `package`
+  method, which populates the CompiledMethod with information from the
+  Generator object.
+
+## Customization
+
+In general, the `package` method is designed to populate the
+CompiledMethod with a group of variables. However, you could also use
+the packager to populate another object with the same interface.
+However, it would not necessarily be useful on its own, without
+additional customizations later on

web/doc/en/bytecode-compiler/writer.markdown

@@ -0,0 +1,42 @@
+---
+layout: doc_en
+title: Writer Stage
+previous: Packager Stage
+previous_url: bytecode-compiler/packager
+next: Transformations
+next_url: bytecode-compiler/transformations
+---
+
+Once the Packager has created the CompiledMethod, Rubinius will write
+the method to a file for future consumption. For instance, after a file
+is required for the first time, subsequent requires will load the file
+from disk, rather than loading in the Ruby code, then parsing and
+compiling it.
+
+This stage is extremely simple. It takes the original input file name,
+appends `c` to the end, and calls Rubinius::CompiledFile.dump with the
+CompiledMethod from the previous stage and the file name to write to.
+
+Once it finishes writing the file to the disk, it returns the input (the
+CompiledMethod), which becomes the return value of the entire compiler
+process.
+
+## Files Referenced
+
+* *lib/compiler/compiled_file.rb*: the CompiledFile implementation.
+  `CompiledFile.dump` is called in order to perform the actual dumping.
+
+## Customizing
+
+This stage is actually optional, and is only used when compiling a file.
+When compiling a String, such as with eval, this step is skipped. In
+that case, the compiler stops in the Packager stage, and returns the
+CompiledMethod from that stage as the return value of the compiler.
+
+Because of the architecture of the Rubinius compiler, it is easy to add
+additional stages to the end of the process, and as long as each of
+those stages returns the inputted CompiledMethod (or a different
+CompiledMethod), everything will work as expected.
+
+For more information, please read [Customizing the Compiler
+Pipeline](/bytecode-compiler/cutomization/).

web/doc/en/index.markdown

@@ -34,11 +34,15 @@ next_url: what-is-rubinius
     1. [Instructions](/doc/en/virtual-machine/instructions/)
     1. [Custom Dispatch Logic](/doc/en/virtual-machine/custom-dispatch-logic/)
 1. [Bytecode Compiler](/doc/en/bytecode-compiler/)
-    1. [Parser](/doc/en/bytecode-compiler/parser/)
+    1. [Parser Stage](/doc/en/bytecode-compiler/parser/)
     1. [AST](/doc/en/bytecode-compiler/ast/)
-    1. [Compiler](/doc/en/bytecode-compiler/compiler/)
+    1. [Generator Stage](/doc/en/bytecode-compiler/generator/)
+    1. [Encoder Stage](/doc/en/bytecode-compiler/encoder/)
+    1. [Packager Stage](/doc/en/bytecode-compiler/packager/)
+    1. [Writer Stage](/doc/en/bytecode-compiler/writer/)
+    1. Printers
     1. [Transformations](/doc/en/bytecode-compiler/transformations/)
-    1. [Generator](/doc/en/bytecode-compiler/generator/)
+    1. [Customizing the Pipeline](/doc/en/bytecode-compiler/customization/)
 1. [JIT Compiler](/doc/en/jit/)
 1. [Garbage Collector](/doc/en/garbage-collector/)
     1. [Nursery](/doc/en/garbage-collector/nursery/)