@@ -1,738 +1,721 @@ -Introduction to the Rubinius compiler -===================================== += Introduction to the Rubinius compiler This document presents the sequence of transformations that it takes to compile a very simple Ruby source file into Rubinius bytecode (or, to be specific, intcode) for the purposes of understanding the basic -operation of the compiler component of the VM. Even source files with -more complex syntax will follow the same general processing pattern. +operation of the compiler component of the VM. Even source files with +more complex syntax will follow the same general processing pattern. The compiler is conceptually very straightforward and elegant. I use a very liberal pseudocode syntax for most of the explanation just to keep things simple. For any clarifications, you can check the source. Generally any data structures shown represent what is being passed to the next step rather than the state in the explanation just -above it: for example in the first stage, the sexp shown is always +above it: for example in the first stage, the sexp shown is always essentially the "argument" to the next method or section. - - -Stage 0: Input -=========================== +== Stage 0: Input The sample input is in test.rb and looks like this: - def foo() - puts "foo" - end + def foo() + puts "foo" + end -Rubinius extends `File` with the method `.to_sexp` which will have +Rubinius extends File with the method #to_sexp which will have the parser produce us a nested set of Arrays with the necessary info -to be able to interpret (or in our case, compile) the program. How -the parser works is outside the scope of treatise but this is its +to be able to interpret (or in our case, compile) the program. How +the parser works is outside the scope of treatise but this is its output, hereafter referred to as the *sexp* (S-expression): - [:newline, 1, "test.rb", - [:defn, :foo, - [:scope, - [:block, [:args], - [:newline, 2, "test.rb", - [:fcall, :puts, [:array, [:str, "foo"]]] - ] - ], - [] - ] + [:newline, 1, "test.rb", + [:defn, :foo, + [:scope, + [:block, [:args], + [:newline, 2, "test.rb", + [:fcall, :puts, [:array, [:str, "foo"]]] + ] + ], + [] ] ] + ] -If you compare the sexp to the code snippet above, is pretty simple. -`:defn` is clearly the method definition, `:scope` and `:block` contain -the code within it and that contained code is the `:fcall`. The -`:newline`s are just that, line information. This all will become +If you compare the sexp to the code snippet above, is pretty simple. ++:defn+ is clearly the method definition, +:scope+ and +:block+ contain +the code within it and that contained code is the +:fcall+. The ++:newline+ nodes are just that, line information. This all will become much clearer as we go along. -Next step is to create a new `Compiler` instance (nothing fancy there) +Next step is to create a new Compiler instance (nothing fancy there) and get to work translating +== Stage 1: Sexp to AST -Stage 1: Sexp to AST -========================= +=== 1.1 Descending into the sexp -1.1 Descending into the sexp ----------------------------- +The sexp is first sent to <tt>Compiler#into_script</tt> (compiler.rb) which +will eventually return some type of a Node object for further processing. +Nodes represent nodes in the resulting AST (Abstract Syntax Tree). -The sexp is first sent to `Compiler#into_script` (compiler.rb) which will -eventually return some type of a Node object for further processing. `Nodes` -represent nodes in the resulting <acronym title="Abstract Syntax Tree">AST</acronym>. +General notes: -Couple general notes: - -* compiler.rb implements the compiler infrastructure as well as the basic - Node class and its default operations. Subclasses of Node reside in +* compiler.rb implements the compiler infrastructure as well as the basic + Node class and its default operations. Subclasses of Node reside in nodes.rb and override methods as necessary. - -* Node.kind allows node types to register themselves into a mapping of sexp - node name => node class, this is `Compiler::Node::Mapping`. For example, - a method definition appears as `:defn` in the sexp and that is mapped to - the `Define` node although usually the names are the same with the exception - that the node class name is of course capitalised. - +* Node.kind allows node types to register themselves into a mapping of sexp + node name => node class, this is <tt>Compiler::Node::Mapping</tt>. For + example, a method definition appears as +:defn+ in the sexp and that is + mapped to the Define node although usually the names are the same with the + exception that the node class name is of course capitalised. * Error handling is present all along the way but I ignore it because there is nothing fancy there. Any errors here are unrecoverable. +Compiler#into_script wraps the plain sexp inside a +:script+ and sends +it to Compiler#convert_sexp. -`Compiler#into_script` wraps the plain sexp inside a `:script` and sends -it to `Compiler#convert_sexp`. - - [:script, - [:newline, 1, "test.rb", - [:defn, :foo, - [:scope, - [:block, [:args], - [:newline, 2, "test.rb", - [:fcall, :puts, [:array, [:str, "foo"]]] - ] - ], - [] - ] + [:script, + [:newline, 1, "test.rb", + [:defn, :foo, + [:scope, + [:block, [:args], + [:newline, 2, "test.rb", + [:fcall, :puts, [:array, [:str, "foo"]]] + ] + ], + [] ] ] ] + ] -We will run into `#convert_sexp` pretty much every single step of the way. -It simply looks up the correct node type from the mapping by using the -first element in the sexp that it receives and then calls the `.create` +We will run into #convert_sexp pretty much every single step of the way. +It simply looks up the correct node type from the mapping by using the +first element in the sexp that it receives and then calls the #create method on the class with the sexp and a reference to the compiler. -So `c.convert_sexp sexp` -> `Script.create c, sexp` - - [:script, - [:newline, 1, "test.rb", - [:defn, :foo, - [:scope, - [:block, [:args], - [:newline, 2, "test.rb", - [:fcall, :puts, [:array, [:str, "foo"]]] - ] - ], - [] - ] +So <tt>c.convert_sexp sexp</tt> -> <tt>Script.create c, sexp</tt> + + [:script, + [:newline, 1, "test.rb", + [:defn, :foo, + [:scope, + [:block, [:args], + [:newline, 2, "test.rb", + [:fcall, :puts, [:array, [:str, "foo"]]] + ] + ], + [] ] ] ] + ] - -None of the Node subclasses implement `.create` themselves so we are still -in compiler.rb. `.create` shifts away the now-unnecessary first element of +None of the Node subclasses implement #create themselves so we are still +in compiler.rb. .create shifts away the now-unnecessary first element of the sexp and then creates a new instance of the node subclass. The following step is the main function of this compilation stage. We ask -the Node subclass instance to `#consume` the remaining sexp which produces -an output. By default `#consume` will simply recursively process all -nested Arrays (sexps) and leave any non-Array elements alone but this -behaviour is overridden by some subclasses (as we will see shortly) to +the Node subclass instance to #consume the remaining sexp which produces +an output. By default #consume will simply recursively process all +nested Arrays (sexps) and leave any non-Array elements alone but this +behaviour is overridden by some subclasses (as we will see shortly) to make the most important processing happen. -The output is then sent to the `#args` method of the node instance or, -if the node type needs argument normalisation (to reconcile all variants -of a particular sexp type), to `#normalize` instead. Neither of these -have a default implementation in `Node` so look under the specific class +The output is then sent to the #args method of the node instance or, +if the node type needs argument normalisation (to reconcile all variants +of a particular sexp type), to #normalize instead. Neither of these +have a default implementation in Node so look under the specific class in nodes.rb. -In our case, `Script.create c, sexp`, we see the first slight modification -in creating the `Script` instance. `Script < ClosedScope`, and the purpose of -all `ClosedScopes` is to represent visibility scopes in the code. The main -part here is that a new `LocalScope` object is created (you can see locals.rb +In our case, Script.create c, sexp, we see the first slight modification +in creating the Script instance. Script < ClosedScope, and the purpose of +all ClosedScopes is to represent visibility scopes in the code. The main +part here is that a new LocalScope object is created (you can see locals.rb but we will get back to this later.) -Next, we step into `Script#consume` (or `ClosedScope#consume` to be +Next, we step into Script#consume (or ClosedScope#consume to be specific) with our newly stripped sexp. - # Script.create c, sexp - # Script.new (creates a new LocalScope) - # s.consume sexp - # args *result - [ - [:newline, 1, "test.rb", - [:defn, :foo, - [:scope, - [:block, [:args], - [:newline, 2, "test.rb", - [:fcall, :puts, [:array, [:str, "foo"]]] - ] - ], - [] - ] + # Script.create c, sexp + # Script.new (creates a new LocalScope) + # s.consume sexp + # args *result + [ + [:newline, 1, "test.rb", + [:defn, :foo, + [:scope, + [:block, [:args], + [:newline, 2, "test.rb", + [:fcall, :puts, [:array, [:str, "foo"]]] + ] + ], + [] ] ] ] + ] -`ClosedScope#consume` assumes that it is getting a single-element +ClosedScope#consume assumes that it is getting a single-element Array containing the rest of the sexp for this particular scope (in our case the entire script.) It has a couple other important duties too, though: firstly, it -uses the compiler's `#set` method to set the current scope as itself -as well as the default visibility as `public`. `#set` and `#get` are -the compiler's way to change and describe the current compilation -state and various auxiliary data. +uses the compiler's #set method to set the current scope as itself +as well as the default visibility as public. #set and #get are +the compiler's way to change and describe the current compilation +state and various auxiliary data. -Secondly, the scope will `#formalize!` all contained scopes. We -will return to this later, but formalization is the process of +Secondly, the scope will #formalize! all contained scopes. We +will return to this later, but formalization is the process of reserving stack space and assigning indexes to any local variables or arguments. - # Script#consume (nodes.rb) - # super to ClosedScope#consume - # out = convert sexp.first (which is a Newline) - # formalize all scopes - [:newline, 1, "test.rb", - [:defn, :foo, - [:scope, - [:block, [:args], - [:newline, 2, "test.rb", - [:fcall, :puts, [:array, [:str, "foo"]]] - ] - ], - [] - ] + # Script#consume (nodes.rb) + # super to ClosedScope#consume + # out = convert sexp.first (which is a Newline) + # formalize all scopes + [:newline, 1, "test.rb", + [:defn, :foo, + [:scope, + [:block, [:args], + [:newline, 2, "test.rb", + [:fcall, :puts, [:array, [:str, "foo"]]] + ] + ], + [] ] ] + ] -The `Node#convert` call you see there just calls `Compiler#convert_sexp` +The Node#convert call you see there just calls Compiler#convert_sexp which, as you recall, is where this whole thing got started. This will be a recurring pattern. -Through the normal route, we get to `Newline.create` which in turn -uses default processing to `#consume` the rest of the sexp (the first +Through the normal route, we get to Newline.create which in turn +uses default processing to #consume the rest of the sexp (the first element is again stripped after the node type lookup.) - # Compiler#convert_sexp -> Newline.create - # Newline#consume sexp.shift - [1, "test.rb", - [:defn, :foo, - [:scope, - [:block, [:args], - [:newline, 2, "test.rb", - [:fcall, :puts, [:array, [:str, "foo"]]] - ] - ], - [] - ] + # Compiler#convert_sexp -> Newline.create + # Newline#consume sexp.shift + [1, "test.rb", + [:defn, :foo, + [:scope, + [:block, [:args], + [:newline, 2, "test.rb", + [:fcall, :puts, [:array, [:str, "foo"]]] + ] + ], + [] ] ] + ] -`Newline#consume` first registers the current file and line +Newline#consume first registers the current file and line information (the first two elements) with the compiler to help -pinpoint compilation errors and then falls back on default -processing from `Node#consume`. +pinpoint compilation errors and then falls back on default +processing from Node#consume. -The process here is simple: loop through the sexp and push -elements to output. All nested sexps are recursively `#converted` +The process here is simple: loop through the sexp and push +elements to output. All nested sexps are recursively #converted first. -So here the eventual output to be returned to `.create` is -just a three-element Array: `[1, 'test.rb', <Define node>]`. +So here the eventual output to be returned to .create is +just a three-element Array: [1, 'test.rb', <Define node>]. -As we can see, the `:defn` node will need processing so we -move on to that. +As we can see, the +:defn+ node will need processing so we +move on to that. - # convert [:defn - [:defn, :foo, - [:scope, - [:block, [:args], - [:newline, 2, "test.rb", - [:fcall, :puts, [:array, [:str, "foo"]]] - ] - ], - [] - ] + # convert [:defn + [:defn, :foo, + [:scope, + [:block, [:args], + [:newline, 2, "test.rb", + [:fcall, :puts, [:array, [:str, "foo"]]] + ] + ], + [] ] + ] (I will start abbreviating some common steps here.) - # n.convert -> c.convert_sexp -> Define.create - # Define.new.consume sexp.shift - [:foo, - [:scope, - [:block, [:args], - [:newline, 2, "test.rb", - [:fcall, :puts, [:array, [:str, "foo"]]] - ] - ], - [] - ] + # n.convert -> c.convert_sexp -> Define.create + # Define.new.consume sexp.shift + [:foo, + [:scope, + [:block, [:args], + [:newline, 2, "test.rb", + [:fcall, :puts, [:array, [:str, "foo"]]] + ] + ], + [] ] + ] -`Define` is also a `ClosedScope` but it will significantly augment -the normal `#consume` from the latter. First we separate the name -and the body, after which the body gets supered to `ClosedScope#consume` +Define is also a ClosedScope but it will significantly augment +the normal #consume from the latter. First we separate the name +and the body, after which the body gets supered to ClosedScope#consume wrapped inside a dummy Array. After getting the body back, we will massage it a bit to suit our needs but this will be done in part 2. -The eventual output here will be `[name, <scope>, <args>]`. +The eventual output here will be [name, <scope>, <args>]. # Define#consume -> ClosedScope#consume [body] -> convert body.first - [:scope, - [:block, [:args], - [:newline, 2, "test.rb", + [:scope, + [:block, [:args], + [:newline, 2, "test.rb", [:fcall, :puts, [:array, [:str, "foo"]]] ] - ], + ], [] ] -`Scope` nodes do not manage the scopes themselves, they are merely -abstractions within the actual visibility scopes. `Scope#consume` +Scope nodes do not manage the scopes themselves, they are merely +abstractions within the actual visibility scopes. Scope#consume assumes it is getting a two-element sexp of which only the first needs further processing (the second contains local variable names.) - # n.convert -> c.convert_sexp -> Scope.create -> Scope#consume - [ - [:block, [:args], - [:newline, 2, "test.rb", - [:fcall, :puts, [:array, [:str, "foo"]]] - ] - ], - [] - ] - - # Scope#consume - # sexp[0] = convert sexp[0] - [:block, - [:args], - [:newline, 2, "test.rb", + # n.convert -> c.convert_sexp -> Scope.create -> Scope#consume + [ + [:block, [:args], + [:newline, 2, "test.rb", [:fcall, :puts, [:array, [:str, "foo"]]] ] - ] + ], + [] + ] + + # Scope#consume + # sexp[0] = convert sexp[0] + [:block, + [:args], + [:newline, 2, "test.rb", + [:fcall, :puts, [:array, [:str, "foo"]]] + ] + ] -`Block`, then, typically encapsulates any type of block of code, +Block, then, typically encapsulates any type of block of code, not a lambda block. It is used in method definitions, if-expressions -and so on. +and so on. - # c.convert_sexp -> Block.create - # Block.new.consume sexp.shift - [ - [:args], - [:newline, 2, "test.rb", - [:fcall, :puts, [:array, [:str, "foo"]]] - ] - ] + # c.convert_sexp -> Block.create + # Block.new.consume sexp.shift + [ + [:args], + [:newline, 2, "test.rb", + [:fcall, :puts, [:array, [:str, "foo"]]] + ] + ] This is the first time we encounter a sexp with two separate nested sexps. Both will be processed separately and the output will be just -`[<arg node>, <newline node>]`. - - # Block#consume -> Node#consume - # Loop through sexp, recursively convert nested sexps - # convert [:args - [:args] - # convert [:newline - [:newline, 2, "test.rb", - [:fcall, :puts, [:array, [:str, "foo"]]] - ] +[<arg node>, <newline node>]. + + # Block#consume -> Node#consume + # Loop through sexp, recursively convert nested sexps + # convert [:args + [:args] + # convert [:newline + [:newline, 2, "test.rb", + [:fcall, :puts, [:array, [:str, "foo"]]] + ] Stepping through Arguments first. - # c.convert_sexp -> Arguments.create - # Arguments.new.consume sexp.shift - [] + # c.convert_sexp -> Arguments.create + # Arguments.new.consume sexp.shift + [] Since we have no arguments, there is no work to be done. However, to -make things consistent, `Arguments#consume` will return a structure -representing no args of any kind, `[[], [], nil, nil]` (the fields are +make things consistent, Arguments#consume will return a structure +representing no args of any kind, [[], [], nil, nil] (the fields are required arg names, optional arg names, splat arg name and default arg -computations. See the source in nodes.rb for more info.) +computations. See the source in nodes.rb for more info.) The second node to process is the newline, one of which we have seen -already. Nothing new here. - - # c.convert_sexp -> Newline.create - # Newline.new.consume sexp.shift - [2, "test.rb", - [:fcall, :puts, [:array, [:str, "foo"]]] - ] +already. Nothing new here. - # Newline#consume - # The line and file are passed to Compiler#set_position - # super sexp -> Node#consume sexp - # Loop through sexp, recursively convert nested sexps - # convert [:fcall + # c.convert_sexp -> Newline.create + # Newline.new.consume sexp.shift + [2, "test.rb", [:fcall, :puts, [:array, [:str, "foo"]]] - -`FCall` is a bit more interesting. It represents a method call without + ] + + # Newline#consume + # The line and file are passed to Compiler#set_position + # super sexp -> Node#consume sexp + # Loop through sexp, recursively convert nested sexps + # convert [:fcall + [:fcall, :puts, [:array, [:str, "foo"]]] + +FCall is a bit more interesting. It represents a method call without an explicit receiver ("functional" style) and it is the first node that -must be `#normalize`d. `FCall < Call < MethodCall < Node`. - - # c.convert_sexp -> FCall.create - # FCall.new - # MethodCall does a bit of extra work here by storing - # the scope type (it was `#set` before, if you recall.) - # In our case the scope type is `Script`. We also start - # out assuming no block is involved with this call. - # - # f.consume sexp.shift - [:puts, [:array, [:str, "foo"]]] - - # FCall#consume -> Node#consume - # Loop through sexp, recursively convert nested sexps - # convert [:array - [[:array, [:str, "foo"]]] - - # c.convert_sexp -> ArrayLiteral.create - # ArrayLiteral.new.consume sexp.shift - [[:str, "foo"]] - - # ArrayLiteral#consume -> Node#consume - # Loop through sexp, recursively convert nested sexps - # convert [:str - [:str, "foo"] - - # c.convert_sexp -> StringLiteral.create - # StringLiteral.new.consume sexp.shift - ["foo"] - - # StringLiteral#consume -> Node#consume - # Loop through sexp, recursively convert nested sexps - # (Nothing to do, just "foo" remains.) +must be #normalized. FCall < Call < MethodCall < Node. + + # c.convert_sexp -> FCall.create + # FCall.new + # MethodCall does a bit of extra work here by storing + # the scope type (it was #set before, if you recall.) + # In our case the scope type is Script. We also start + # out assuming no block is involved with this call. + # + # f.consume sexp.shift + [:puts, [:array, [:str, "foo"]]] + + # FCall#consume -> Node#consume + # Loop through sexp, recursively convert nested sexps + # convert [:array + [[:array, [:str, "foo"]]] + + # c.convert_sexp -> ArrayLiteral.create + # ArrayLiteral.new.consume sexp.shift + [[:str, "foo"]] + + # ArrayLiteral#consume -> Node#consume + # Loop through sexp, recursively convert nested sexps + # convert [:str + [:str, "foo"] + + # c.convert_sexp -> StringLiteral.create + # StringLiteral.new.consume sexp.shift + ["foo"] + + # StringLiteral#consume -> Node#consume + # Loop through sexp, recursively convert nested sexps + # (Nothing to do, just "foo" remains.) At this point, we are at the end of the tree. There is nothing more to parse and we are ready for the next phase. +=== 1.2 Creating the AST - -1.2 Creating the AST ------------------------------ - -Now we traverse back up the sexp, pushing the generated Nodes -upwards as we go. As you remember, the second part of each -`.create` was sending the output of `#consume` to `#args` or -`#normalize`. In the last step, we ended up with `StringLiteral` +Now we traverse back up the sexp, pushing the generated Nodes +upwards as we go. As you remember, the second part of each +.create was sending the output of #consume to #args or +#normalize. In the last step, we ended up with StringLiteral and that is where we will pick up. -The output of a default `Node#consume` is basically just the original +The output of a default Node#consume is basically just the original sexp but with any nested sexps also converted (this may not hold -true for subclasses.) In the case of `StringLiteral`, there were -no nested expressions, just the actual literal. (`<-` is used to +true for subclasses.) In the case of StringLiteral, there were +no nested expressions, just the actual literal. (<- is used to denote returning.) -`StringLiteral#consume` returns `["foo"]` and execution picks -back up in `Stringliteral.create` where the returned value is -passed into `StringLiteral#args`. All `#args` does here is -store the string within the `StringLiteral` node. After this, -`StringLiteral.create` is done, returning the node object. +StringLiteral#consume returns ["foo"] and execution picks +back up in Stringliteral.create where the returned value is +passed into StringLiteral#args. All #args does here is +store the string within the StringLiteral node. After this, +StringLiteral.create is done, returning the node object. -The execution then unwinds from `StringLiteral.create` to -`Compiler#convert_sexp` to `<Node class>#convert` and ends up -in the `#consume` of the parent. In the case of `StringLiteral`, -we end up in `ArrayLiteral#consume`. +The execution then unwinds from StringLiteral.create to +Compiler#convert_sexp to <Node class>#convert and ends up +in the #consume of the parent. In the case of StringLiteral, +we end up in ArrayLiteral#consume. This sequence will largely be the same for all cases so you will see it abbreviated like this: - # StringLiteral#consume - # <- StringLiteral.create - # StringLiteral#args return - # `string = 'foo' - # <- c.convert_sexp - # <- n.convert - # <- ArrayLiteral#consume - - StringLiteral - .string = 'foo' - -So the `StringLiteral` goes back to `ArrayLiteral#consume` -which then sends it to `ArrayLiteral#args`: - - # ArrayLiteral#consume - # <= ArrayLiteral.create - # ArrayLiteral#args - # `body = - - ArrayLiteral - .body = [StringLiteral - .string = 'foo'] - - # Return to FCall#consume - -`FCall#consume` yields its process results back to `.create` -and, because `FCall` does implement `#normalize`, the output -is sent there rather than `#args`. `#collapse_args` will just -strip away the unnecessary `ArrayLiteral` and give us an -internal Array containing the `ArrayLiteral` contents instead. -`FCall` also checks `#detect_special_form` to be able to -translate pseudo-calls like `:ivar_as_index` but we have -none in our little script. - - # FCall#consume - out = [:puts, anArrayLiteral] - <- FCall.create - FCall#normalize - `method, `arguments = :puts, al - collapse_args -> Call#collapse_args - - FCall - .method = :puts - .arguments = ArrayLiteral - ... + # StringLiteral#consume + # <- StringLiteral.create + # StringLiteral#args return + # string = 'foo' + # <- c.convert_sexp + # <- n.convert + # <- ArrayLiteral#consume + + StringLiteral + .string = 'foo' - # collapse_args +So the StringLiteral goes back to ArrayLiteral#consume +which then sends it to ArrayLiteral#args: - FCall - .method = :puts - .arguments = [StringLiteral - .string = 'foo'] + # ArrayLiteral#consume + # <= ArrayLiteral.create + # ArrayLiteral#args + # body = + + ArrayLiteral + .body = [StringLiteral + .string = 'foo'] + + # Return to FCall#consume + +FCall#consume yields its process results back to .create +and, because FCall does implement #normalize, the output +is sent there rather than #args. #collapse_args will just +strip away the unnecessary ArrayLiteral and give us an +internal Array containing the ArrayLiteral contents instead. +FCall also checks #detect_special_form to be able to +translate pseudo-calls like +:ivar_as_index+ but we have +none in our little script. - # Return to Newline#consume + # FCall#consume + out = [:puts, anArrayLiteral] + <- FCall.create + FCall#normalize + method, arguments = :puts, al + collapse_args -> Call#collapse_args + + FCall + .method = :puts + .arguments = ArrayLiteral + ... + + # collapse_args + + FCall + .method = :puts + .arguments = [StringLiteral + .string = 'foo'] + + # Return to Newline#consume -Nothing special in `Newline`, the processed nodes just get +Nothing special in Newline, the processed nodes just get stored. Notably we do not even clear the file, line info that was set earlier. - # Newline#consume - # out = [1, 'test.rb', fc] - # <- Newline.create - # We do NOT clear the compiler file, line positions - # Newline#args - # `file, `line, `child = 'test.rb', 1, fc - - Newline - .file = 'test.rb' - .line = 1 - .child = FCall - .method = :puts - .arguments = [StringLiteral - .string = 'foo'] + # Newline#consume + # out = [1, 'test.rb', fc] + # <- Newline.create + # We do NOT clear the compiler file, line positions + # Newline#args + # file, line, child = 'test.rb', 1, fc + + Newline + .file = 'test.rb' + .line = 1 + .child = FCall + .method = :puts + .arguments = [StringLiteral + .string = 'foo'] There were two forks in Block#consume so we complete the other -one, `Arguments`, before going further up the chain. We have +one, Arguments, before going further up the chain. We have no arguments so your basic assignment again. (If we did have -args, `Arguments#populate` would handle properly associating -the variable names to `Local` objects.) - - # Arguments#consume - # out = [[], [], nil, nil] - # <- Arguments#create - # Arguments#args - # `required, `optional, `splat, `defaults = [], [], nil, nil - # `block_arg = nil (may be set later) - # Skip Arguments#populate - - Arguments - .required = [] - .optional = [] - .splat = nil - .defaults = nil - .block_arg = nil - -Alright, now we have both trees back and can finish up with -the `Block` which actually just stores both the args *and* +args, Arguments#populate would handle properly associating +the variable names to Local objects.) + + # Arguments#consume + # out = [[], [], nil, nil] + # <- Arguments#create + # Arguments#args + # required, optional, splat, defaults = [], [], nil, nil + # block_arg = nil (may be set later) + # Skip Arguments#populate + + Arguments + .required = [] + .optional = [] + .splat = nil + .defaults = nil + .block_arg = nil + +Alright, now we have both trees back and can finish up with +the Block which actually just stores both the args *and* the body in a single Array. - # Block#consume - # out = [args, nl1] - # <- Block.create - # Block#args - - Block - .body = [Arguments - .required = [] - .optional = [] - .splat = nil - .defaults = nil - .block_arg = nil - , - Newline - .file = 'test.rb' - .line = 1 - .child = FCall - .method = :puts - .arguments = [StringLiteral - .string = 'foo'] - ] + # Block#consume + # out = [args, nl1] + # <- Block.create + # Block#args + + Block + .body = [Arguments + .required = [] + .optional = [] + .splat = nil + .defaults = nil + .block_arg = nil + , + Newline + .file = 'test.rb' + .line = 1 + .child = FCall + .method = :puts + .arguments = [StringLiteral + .string = 'foo'] + ] + +Back up we go to Scope#consume. As you rememeber, this only +manipulated sexp[0] (which is our Block), so: -Back up we go to `Scope#consume`. As you rememeber, this only -manipulated sexp[0] (which is our `Block`), so: - - # Scope#consume - # out = [Block ..., []] - # <- Scope.create - - Scope - .locals = [] - .block = [Arguments - ..., - Newline - ...] - - Block - .body = [Arguments - .required = [] - .optional = [] - .splat = nil - .defaults = nil - .block_arg = nil - , - Newline - .file = 'test.rb' - .line = 1 - .child = FCall - .method = :puts - .arguments = [StringLiteral - .string = 'foo'] - ] - -Return gets us to to `ClosedScope#consume` because `Define#consume` + # Scope#consume + # out = [Block ..., []] + # <- Scope.create + + Scope + .locals = [] + .block = [Arguments + ..., + Newline + ...] + + Block + .body = [Arguments + .required = [] + .optional = [] + .splat = nil + .defaults = nil + .block_arg = nil + , + Newline + .file = 'test.rb' + .line = 1 + .child = FCall + .method = :puts + .arguments = [StringLiteral + .string = 'foo'] + ] + +Return gets us to to ClosedScope#consume because Define#consume supered to it for its body part. -`ClosedScope#consume` is done with its subnodes so it will `#formalize!` +ClosedScope#consume is done with its subnodes so it will #formalize! all scopes. This deals with assigning indexes for local variables, and possibly reserving stack space when necessary. We have no locals so -we need not worry about it. The `Scope` from above is returned to -`Define#consume`. - -`Define#consume` extracts `Scope.block.body` back out again and then -further splits it into args and the rest. If a block argument is +we need not worry about it. The Scope from above is returned to +Define#consume. + +Define#consume extracts Scope.block.body back out again and then +further splits it into args and the rest. If a block argument is detected at this point, it gets recorded in the args. - # Define#consume - # out = [:foo, <modified scope>, <args extracted from scope>] - # <- Define.create - # Define#args - - # This... - - Scope - .locals = [] - .block = Block - .body = [Arguments - .required = [] - .optional = [] - .splat = nil - .defaults = nil - .block_arg = nil - , - Newline - .file = 'test.rb' - .line = 1 - .child = FCall - .method = :puts - .arguments = [StringLiteral - .string = 'foo'] - ] - - # Becomes this when we extract scope.block.args - # And add the Define node - - Define - .name = :foo - .body = Scope - .block = Block - .body = [Newline - .file = 'test.rb' - .line = 2 - .child = FCall - .method = :puts - .arguments = [StringLiteral - .string = 'foo']] - .locals = [] - .args = Arguments # args are here now - .required = [] - .optional = [] - .splat = nil - .defaults = nil - .block_arg = nil - - -`Newline#consume` has nothing new to offer, we just store the + # Define#consume + # out = [:foo, <modified scope>, <args extracted from scope>] + # <- Define.create + # Define#args + + # This... + + Scope + .locals = [] + .block = Block + .body = [Arguments + .required = [] + .optional = [] + .splat = nil + .defaults = nil + .block_arg = nil + , + Newline + .file = 'test.rb' + .line = 1 + .child = FCall + .method = :puts + .arguments = [StringLiteral + .string = 'foo'] + ] + + # Becomes this when we extract scope.block.args + # And add the Define node + + Define + .name = :foo + .body = Scope + .block = Block + .body = [Newline + .file = 'test.rb' + .line = 2 + .child = FCall + .method = :puts + .arguments = [StringLiteral + .string = 'foo']] + .locals = [] + .args = Arguments # args are here now + .required = [] + .optional = [] + .splat = nil + .defaults = nil + .block_arg = nil + + +Newline#consume has nothing new to offer, we just store the positional information. - - # Newline#consume - # out = [1, 'foo.rb', meth] - # <- Newline.create - # Newline#args - - Newline - .file = 'test.rb' - .line = 1 - .child = Define - .name = :foo - .body = Scope - .block = Block - .body = [Newline - .file = 'test.rb' - .line = 2 - .child = FCall - .method = :puts - .arguments = [StringLiteral - .string = 'foo']] - .locals = [] - .args = Arguments - .required = [] - .optional = [] - .splat = nil - .defaults = nil - .block_arg = nil - -Finally, we are back in `Script#consume`. Being a `ClosedScope`, -it will also `#formalize!` its locals (and we still have none.) - -Curiously, the "body" of the script is just the single newline -but this is just because the top-level is only really concerned -about the code it directly contains. If our original code had been: - def foo - puts "foo" - end + # Newline#consume + # out = [1, 'foo.rb', meth] + # <- Newline.create + # Newline#args + + Newline + .file = 'test.rb' + .line = 1 + .child = Define + .name = :foo + .body = Scope + .block = Block + .body = [Newline + .file = 'test.rb' + .line = 2 + .child = FCall + .method = :puts + .arguments = [StringLiteral + .string = 'foo']] + .locals = [] + .args = Arguments + .required = [] + .optional = [] + .splat = nil + .defaults = nil + .block_arg = nil + +Finally, we are back in Script#consume. Being a ClosedScope, +it will also #formalize! its locals (and we still have none.) + +Curiously, the "body" of the script is just the single newline +but this is just because the top-level is only really concerned +about the code it directly contains. If our original code had been: - foo + def foo + puts "foo" + end + + foo -Then the `Script` would have ended up with its body a `Block` -that in turn contained two `Newline` objects (with line numbers +Then the Script would have ended up with its body a Block +that in turn contained two Newline objects (with line numbers 1 and 5 respectively.) -Anyway, for now `Script#args` just stores the body. +Anyway, for now Script#args just stores the body. That is it for the compilation phase! This is the complete resulting structure for our tiny program: - Script - .body = Newline - .file = 'test.rb' - .line = 1 - .child = Define - .name = :foo - .body = Scope - .block = Block - .body = [Newline - .file = 'test.rb' - .line = 2 - .child = FCall - .method = :puts - .arguments = [StringLiteral - .string = 'foo']] - .locals = [] - .args = Arguments - .required = [] - .optional = [] - .splat = nil - .defaults = nil - .block_arg = nil - -Now we return to `Compiler#into_script`. - - - -Stage 2: Bytecode generation -============================ - -The benefit of having the nested `Node` structure over the original + Script + .body = Newline + .file = 'test.rb' + .line = 1 + .child = Define + .name = :foo + .body = Scope + .block = Block + .body = [Newline + .file = 'test.rb' + .line = 2 + .child = FCall + .method = :puts + .arguments = [StringLiteral + .string = 'foo']] + .locals = [] + .args = Arguments + .required = [] + .optional = [] + .splat = nil + .defaults = nil + .block_arg = nil + +Now we return to Compiler#into_script. + +== Stage 2: Bytecode generation + +The benefit of having the nested Node structure over the original sexp may not be immediately obvious, but it is the ability of the -`Node` objects to have behaviours, unlike the plain sexp. In other -words instead of having an external entity manage it, the `Node`s +Node objects to have behaviours, unlike the plain sexp. In other +words instead of having an external entity manage it, the Nodes will generate their own code using the knowledge that they have accumulated. To separate AST generation from bytecode generation, -`Node`s originally from `node.rb` are reopened in `bytecode.rb` -to add the `#bytecode` method (and some other stuff.) The VM +Nodes originally from node.rb are reopened in bytecode.rb +to add the #bytecode method (and some other stuff.) The VM works off a stack so the generated bytecode will be stack-based. A word about that code: this stage is a bit ambiguous. Conceptually @@ -740,610 +723,599 @@ speaking the first step is to generate assembly code of sorts, essentially something that is still "symbolic code" and not just a string of ones and zeroes (that would be "machine code".) The benefit of the former is, of course, that it is easier for humans -to read than the raw bytes. We could technically have the `Node`s +to read than the raw bytes. We could technically have the Nodes produce a string of instructions (and in fact that is just what -`TextGenerator` in `text.rb` does) and then interpret that string +TextGenerator in text.rb does) and then interpret that string into the *real* bytecode or machine code but this is a bit wasteful. -Rubinius opts for a sort-of middle ground by first having the `Node` -objects generate a "stream" or sequence of instructions in the form +Rubinius opts for a sort-of middle ground by first having the Node +objects generate a "stream" or sequence of instructions in the form of simpler Ruby objects which will then later be encoded into the raw bytecode format in stage 3. As you have probably noticed, the Rubinius runtime model is based on methods. In fact, any script or snippet is essentially a method -in its own right and the basic building blocks in the VM are -`CompiledMethod`s more on which later and `MethodContext`s which -are beyond the scope of this treatise. For this intermediate -representation we naturally opt for the same approach. +in its own right and the basic building blocks in the VM are +CompiledMethods more on which later and MethodContexts which +are beyond the scope of this treatise. For this intermediate +representation we naturally opt for the same approach. -`MethodDescription` will store our important data. The second and -more important player is the `Generator` class from `generator.rb` +MethodDescription will store our important data. The second and +more important player is the Generator class from generator.rb which, despite its name, does not actually so much generate the code as act as receptacle thereof. In other words, the "syntax," -if you wish, is in `Generator` and the logic comes from the `Node`s. -The `Encoder` will also make an appearance, although its main +if you wish, is in Generator and the logic comes from the Nodes. +The Encoder will also make an appearance, although its main function is not explored until the next stage. -A note about the pseudocode used in the description here: The +A note about the pseudocode used in the description here: The generated code will be in a sort-of-assembly syntax and I will -use assembly comments `;` for any comments that could be imagined -to be left there by the "writer." Ruby comments `#` are used +use assembly comments ; for any comments that could be imagined +to be left there by the "writer." Ruby comments # are used for "meta-information." I will also introduce new pseudocode elements as they are needed so as to not confuse matters up -front and will always explain when I do so. Hopefully this will +front and will always explain when I do so. Hopefully this will help rather than hinder clarity. Now. -If you still remember, the last stage ended with the topmost `Node`, -a `Script` to be specific, was returned from the compiler. This is -fortunate because only `ClosedScope`s implement the `#to_description` +If you still remember, the last stage ended with the topmost Node, +a Script to be specific, was returned from the compiler. This is +fortunate because only ClosedScopes implement the #to_description method that allows us to start this phase. - # Eventually, the topmost `MethodDescription` is returned. - ast = compiler.into_script ... - desc = ast.to_description + # Eventually, the topmost MethodDescription is returned. + ast = compiler.into_script ... + desc = ast.to_description -This will obviously send us into `ClosedScope#to_description` which -you will find in `bytecode.rb`. +This will obviously send us into ClosedScope#to_description which +you will find in bytecode.rb. -This is a bit of a tangled web. We start with three players: the -Script (which we will start calling `$script`) sets up a new `MethodDescription` -which also provides a `Generator` ( <--- SUBJECT TO CHANGE ) which is -exposed to `$script`. Then, `$script` asks the `MethodDescription` to -run using `$script` as its argument. This, in turn will cause the -`Generator` (`$gen`) to run again using `$script` as its starting +This is a bit of a tangled web. We start with three players: the +Script (which we will start calling $script) sets up a new MethodDescription +which also provides a Generator ( <--- SUBJECT TO CHANGE ) which is +exposed to $script. Then, $script asks the MethodDescription to +run using $script as its argument. This, in turn will cause the +Generator ($gen) to run again using $script as its starting point and as before, starts its descent from here. Eventually after -this the `MethodDescription` gathers `#argument_info` from the `Node` -and returns. Because $script is sort of a pseudomethod, it will +this the MethodDescription gathers #argument_info from the Node +and returns. Because $script is sort of a pseudomethod, it will only have dummy values here. For now, let us proceed though. - desc.run $script -> $gen.run $script + desc.run $script -> $gen.run $script -`Generator#run` is an exceedingly simple matter: it merely requests -the `Node` given to it to execute its `#bytecode` method, passing +Generator#run is an exceedingly simple matter: it merely requests +the Node given to it to execute its #bytecode method, passing itself as a parameter in order to continue down the tree. So in -our case, we head into `Script#bytecode`. +our case, we head into Script#bytecode. -The `#bytecode` methods are the backbone of this entire operation, -as mentioned earlier. Each `Node` will know the logic required to -compile itself and this logic collected by the `#bytecode` methods -calling various methods on the `Generator` object they are passed. -The `Generator`, then, records those operations into its internal +The #bytecode methods are the backbone of this entire operation, +as mentioned earlier. Each Node will know the logic required to +compile itself and this logic collected by the #bytecode methods +calling various methods on the Generator object they are passed. +The Generator, then, records those operations into its internal instruction stream. -`Script#bytecode` starts by generating a prelude (compiler term for +Script#bytecode starts by generating a prelude (compiler term for work that is done to set up a method before code inside it starts executing) which simply reserves space for local variables within this scope. The space required has been determined in the earlier -calls to `ClosedScope#formalize!` which was a part of the cycle -for `ClosedScope#consume`. Since there were no locals at the toplevel, -the size we need is 0. - -To actually achieve this, we call `$gen.allocate_stack` which in -turn just calls `Generator#add` with the argument pair `:allocate_stack, 0` -This is, essentially, the "assembly code." - -`Generator#add` handles incrementing its instruction pointer to -account for the new code in the stream (the ip always points to -the last element in the stream) and then looks up the actual numeric -opcode of our "assembly" instruction :allocate_stack. It happens -to currently be 97 but may change at any time so for clarity +calls to ClosedScope#formalize! which was a part of the cycle +for ClosedScope#consume. Since there were no locals at the toplevel, +the size we need is 0. + +To actually achieve this, we call $gen.allocate_stack which in +turn just calls Generator#add with the argument pair +:allocate_stack+, 0 +This is, essentially, the "assembly code." + +Generator#add handles incrementing its instruction pointer to +account for the new code in the stream (the ip always points to +the last element in the stream) and then looks up the actual numeric +opcode of our "assembly" instruction +:allocate_stack+. It happens +to currently be 97 but may change at any time so for clarity our pseudocode will not refer to it at all, it is just extra -info kept in the stream by the `Generator` itself. +info kept in the stream by the Generator itself. -For our code, the instruction pointer (ip) is not that important -but it is very much so when dealing with branching. After you are +For our code, the instruction pointer (ip) is not that important +but it is very much so when dealing with branching. After you are done with this, see (TODO: link to text about labels) for more info. -The prelude done, `$script` wants to have its body processed before +The prelude done, $script wants to have its body processed before proceeding with the rest of its code so that the instruction stream is in the correct order. - # $script.bytecode - # $gen.allocate_stack 0 - # $gen.add :allocate_stack, 0 - # Increment instruction pointer, look up opcode - # $script.body.bytecode - - # "Assembly" stream - ; No allocation needed - - # We are starting back up here - Script - .body = Newline - .file = 'test.rb' - .line = 1 - .child = Define - ... - -The body itself is a `Newline` node and the first thing one of -those wants to do is to let the `Generator` know its line and -file information. `Generator#set_line` actually does a bit more + # $script.bytecode + # $gen.allocate_stack 0 + # $gen.add :allocate_stack, 0 + # Increment instruction pointer, look up opcode + # $script.body.bytecode + + # "Assembly" stream + ; No allocation needed + + # We are starting back up here + Script + .body = Newline + .file = 'test.rb' + .line = 1 + .child = Define + ... + +The body itself is a Newline node and the first thing one of +those wants to do is to let the Generator know its line and +file information. Generator#set_line actually does a bit more work because for all the lines in the code, it keeps track of the starting and ending instruction pointers (or in other words, stream positions.) Importantly, this information is not recorded -in the instruction stream and is just kept track of by `$gen`. +in the instruction stream and is just kept track of by $gen. (These helper methods are collectively referred to as "commands" whereas the ones that actually affect the stream are "operations.") -Since we just started, line 1 of the real code starts at `$gen.ip == 2` +Since we just started, line 1 of the real code starts at $gen.ip == 2 and ends.. well, we do not know yet. The file is also stored but -since Ruby code is always compiled in units of one file, there is -nothing fancy to do there. +since Ruby code is always compiled in units of one file, there is +nothing fancy to do there. -The only other thing for a `Newline` to do is to send the code +The only other thing for a Newline to do is to send the code it contains to be processed. - # Nothing new in the stream - ; No allocation needed - + # Nothing new in the stream + ; No allocation needed + + Script + .body = Newline + .file = 'test.rb' + .line = 1 + .child = Define + .name = :foo + .body = Scope + ... + .args = Arguments + ... - Script - .body = Newline - .file = 'test.rb' - .line = 1 - .child = Define - .name = :foo - .body = Scope - ... - .args = Arguments - ... - The next step is quite interesting. As you remember, we are technically -inside a `MethodDescription` object--so what happens when we need to +inside a MethodDescription object--so what happens when we need to describe another method? There are some important concepts to cover. -`Define#bytecode` starts off with `$gen.push_literal compile_body()`. -`Define#compile_body` actually produces a new `MethodDescription` -- +Define#bytecode starts off with $gen.push_literal compile_body(). +Define#compile_body actually produces a new MethodDescription -- to which we will get in a moment -- so the "literal" is not one in the sense of a string literal or an array literal. What actually happens -here is that the object given to `#push_literal` is stored with other -literals in `$gen` and its index within that group is used for the +here is that the object given to #push_literal is stored with other +literals in $gen and its index within that group is used for the actual "assembly". The actual literal objects will eventually get -stored in the `CompiledMethod` object (if we are compiling to a file, +stored in the CompiledMethod object (if we are compiling to a file, then even further along they will be marshalled along with the rest -of the `CompiledMethod`.) +of the CompiledMethod.) -Again, in the interest of maintaining the correct sequence, the code +Again, in the interest of maintaining the correct sequence, the code that makes up the method body will get generated first. Exploring -`Define#compile_body` shows a very similar process to what got us -started with `$script`. We set up a new `MethodDescription` which -means that for the time being we will be working with a different -`Generator`. A new prelude is produced but since we still have no -local variables, a 0 stack allocation is needed. - -In the pseudocode, in order to distinguish the `Generator`s, the -original will be `$gen` and the one for `foo` is `$subgen` whenever +Define#compile_body shows a very similar process to what got us +started with $script. We set up a new MethodDescription which +means that for the time being we will be working with a different +Generator. A new prelude is produced but since we still have no +local variables, a 0 stack allocation is needed. + +In the pseudocode, in order to distinguish the Generators, the +original will be $gen and the one for foo is $subgen whenever needed. For fun, I will also indent foo. Next, in this order, we generate the argument handling code and then the method body. - # Toplevel + # Toplevel + ; No allocation needed + + # foo ; No allocation needed - - # foo - ; No allocation needed - - - Script - .body = Newline - .file = 'test.rb' - .line = 1 - .child = Define - .name = :foo - .body = Scope - ... - .args = Arguments - .required = [] - .optional = [] - .splat = nil - .defaults = nil - .block_arg = nil - -The processing of `Arguments` is a bit more involved than our -previous `Node`s (or would be if `#foo` took any.) Even in our -case, though, the argument count needs to be checked so we -figure out how many are needed (0) and how many can be given -(also 0.) Interestingly, the maximum possible number of arguments -is currently 1024. First, back to `Define` and then down the + + Script + .body = Newline + .file = 'test.rb' + .line = 1 + .child = Define + .name = :foo + .body = Scope + ... + .args = Arguments + .required = [] + .optional = [] + .splat = nil + .defaults = nil + .block_arg = nil + +The processing of Arguments is a bit more involved than our +previous Nodes (or would be if #foo took any.) Even in our +case, though, the argument count needs to be checked so we +figure out how many are needed (0) and how many can be given +(also 0.) Interestingly, the maximum possible number of arguments +is currently 1024. First, back to Define and then down the other fork to its body. - # Toplevel - ; No allocation needed - - # foo - ; Argument handling - check_argcount 0 0 ; ip == 3 - - - Script - .body = Newline - .file = 'test.rb' - .line = 1 - .child = Define - .name = :foo - .body = Scope - .block = Block - ... - .locals = [] - - .args = Arguments - ... - -`Define` wants to generate the code for its body so `Scope#bytecode` -is our next destination but all there is to do is to further descend -in and ask for `Block#bytecode` from the block contained (nope, we -are STILL not touching the locals.) `Block` in our case also ends up -just asking its `body` for its `#bytecode`s since we only have one + # Toplevel + ; No allocation needed + + # foo + ; Argument handling + check_argcount 0 0 ; ip == 3 + + Script + .body = Newline + .file = 'test.rb' + .line = 1 + .child = Define + .name = :foo + .body = Scope + .block = Block + ... + .locals = [] + + .args = Arguments + ... + +Define wants to generate the code for its body so Scope#bytecode +is our next destination but all there is to do is to further descend +in and ask for Block#bytecode from the block contained (nope, we +are STILL not touching the locals.) Block in our case also ends up +just asking its body for its #bytecodes since we only have one element in it. - # Toplevel - ; No allocation needed - - # foo - ; Argument handling - check_argcount 0 0 ; ip == 3 - - - Script - .body = Newline - .file = 'test.rb' - .line = 1 - .child = Define - .name = :foo - .body = Scope - .block = Block - .body = [Newline - .file = 'test.rb' - .line = 2 - .child = FCall - ... - .locals = [] - - .args = Arguments - ... - -Another `Newline`! We started one earlier but since that one resides -in `$gen`, opening a new one here really does not have any effect on + # Toplevel + ; No allocation needed + + # foo + ; Argument handling + check_argcount 0 0 ; ip == 3 + + Script + .body = Newline + .file = 'test.rb' + .line = 1 + .child = Define + .name = :foo + .body = Scope + .block = Block + .body = [Newline + .file = 'test.rb' + .line = 2 + .child = FCall + ... + .locals = [] + + .args = Arguments + ... + +Another Newline! We started one earlier but since that one resides +in $gen, opening a new one here really does not have any effect on it. Line 2 starts at ip 3 which gives us the interesting property that ips 0-2 really do not live anywhere. Next, processing goes to the child -of the `Newline`. +of the Newline. - # Toplevel - ; No allocation needed - - # foo - ; Argument handling - check_argcount 0 0 ; ip == 3 - - - Script - .body = Newline - .file = 'test.rb' - .line = 1 - .child = Define - .name = :foo - .body = Scope - .block = Block - .body = [Newline - .file = 'test.rb' - .line = 2 - .child = FCall - .method = :puts - .arguments = [StringLiteral - .string = 'foo']] - .locals = [] - .args = Arguments + # Toplevel + ; No allocation needed + + # foo + ; Argument handling + check_argcount 0 0 ; ip == 3 + + Script + .body = Newline + .file = 'test.rb' + .line = 1 + .child = Define + .name = :foo + .body = Scope + .block = Block + .body = [Newline + .file = 'test.rb' + .line = 2 + .child = FCall + .method = :puts + .arguments = [StringLiteral + .string = 'foo']] + .locals = [] + .args = Arguments The main bytecode generation for all call types happens not in the top -class of that hierarchy, `MethodCall`, but in `Call#bytecode`. So also -for our `FCall`. First order of business is to check whether a plugin +class of that hierarchy, MethodCall, but in Call#bytecode. So also +for our FCall. First order of business is to check whether a plugin can handle this node (TODO: separate doc about the plugin arch) and in this case, the answer is "no" so we proceed. Next, arguments for the -call through `#emit_args`: we do have args and the args are held in +call through #emit_args: we do have args and the args are held in an Array so each of those gets generated -- in reverse order -- and -the argument count is stored. This means a short side trip to -`StringLiteral#bytecode` which just uses `$subgen.push_literal` on its -string and then calls `$subgen.string_dup` which will just cause a new +the argument count is stored. This means a short side trip to +StringLiteral#bytecode which just uses $subgen.push_literal on its +string and then calls $subgen.string_dup which will just cause a new duplicate string to replace the original on the stack when run (here -is a bit of `Generator` magic: any operation that is not explicitly -defined just gets `#add`ed using `#method_missing`.) `#push_literal`, +is a bit of Generator magic: any operation that is not explicitly +defined just gets #added using #method_missing.) #push_literal, as you recall, either stores or retrieves an already-stored literal and gives back its index. A peek at what we look like before heading -back to `FCall#emit_args`. - -(Another pseudocode convention: #<literal> means the index of the -literal from `#find_literal`.) +back to FCall#emit_args. - # Toplevel - ; No allocation needed - - # foo - ; Argument handling - check_argcount 0 0 ; ip == 3 +(Another pseudocode convention: #<literal> means the index of the +literal from #find_literal.) - ; FCall arguments - push_literal #<"foo"> ; ip == 5 - string_dup ; ip == 6 + # Toplevel + ; No allocation needed + + # foo + ; Argument handling + check_argcount 0 0 ; ip == 3 + + ; FCall arguments + push_literal #<"foo"> ; ip == 5 + string_dup ; ip == 6 -Nothing further to do in `#emit_args` but to store the arg count -and then we drop down to `FCall#bytecode` again. There are checks +Nothing further to do in #emit_args but to store the arg count +and then we drop down to FCall#bytecode again. There are checks for block code generation as well as dynamic arguments (these are -`:splat`s, `:argscat`s and `:argspush`es) neither of which affects ++:splats+, +:argscats+ and +:argspushes+) neither of which affects us. Next code to generate would be for the receiver (the receiver could for example itself be a result of a method call) but we are -again lucky because an `FCall` by definition has no receiver so -all calls will be to `self`. This work is all put together in -the `$subgen.send` operation, which in our case will start by setting +again lucky because an FCall by definition has no receiver so +all calls will be to self. This work is all put together in +the $subgen.send operation, which in our case will start by setting the call flags to allow for private methods (which MUST be called -without an explicit receiver*.) Further, the method name we want +without an explicit receiver*.) Further, the method name we want is looked up or stored in literals and its index retrieved. We -also encounter a small optimisation here: normally, the next -instruction would be `:send_stack`, but because we have <5 args, -1 to be precise, a special instruction `:meta_send_stack_1` is +also encounter a small optimisation here: normally, the next +instruction would be +:send_stack+, but because we have <5 args, +1 to be precise, a special instruction +:meta_send_stack_1+ is used instead. This is what we have when we head back towards the top again: - # Toplevel - ; No allocation needed - - # foo - ; Argument handling - check_argcount 0 0 ; ip == 3 - - ; FCall arguments - push_literal #<"foo"> ; ip == 5 - string_dup ; ip == 6 - - ; FCall receiver - push_self ; ip == 7 - - ; FCall send - set_call_flags 1 ; ip == 9 - meta_send_stack_1 :puts ; ip == 11 - - - Script - .body = Newline - .file = 'test.rb' - .line = 1 - .child = Define - .name = :foo - .body = Scope - .block = Block - .body = [Newline - ... - .locals = [] - .args = Arguments - .required = [] - .optional = [] - .splat = nil - .defaults = nil - .block_arg = nil - -Nothing further to do in `Newline#bytecode`, `Block#bytecode` is done -as is `Scope#bytecode` so we finally just end up in `Define#compile_body` -which we need to finish up. Two small things to do, `$subgen.sret` will -emit code necessary to return from the method and `$subgen.close` is a + # Toplevel + ; No allocation needed + + # foo + ; Argument handling + check_argcount 0 0 ; ip == 3 + + ; FCall arguments + push_literal #<"foo"> ; ip == 5 + string_dup ; ip == 6 + + ; FCall receiver + push_self ; ip == 7 + + ; FCall send + set_call_flags 1 ; ip == 9 + meta_send_stack_1 :puts ; ip == 11 + + Script + .body = Newline + .file = 'test.rb' + .line = 1 + .child = Define + .name = :foo + .body = Scope + .block = Block + .body = [Newline + ... + .locals = [] + .args = Arguments + .required = [] + .optional = [] + .splat = nil + .defaults = nil + .block_arg = nil + +Nothing further to do in Newline#bytecode, Block#bytecode is done +as is Scope#bytecode so we finally just end up in Define#compile_body +which we need to finish up. Two small things to do, $subgen.sret will +emit code necessary to return from the method and $subgen.close is a command to finalise any open lines so now we know that line 2 started at ip 3 and ended at ip 12. - # Toplevel - ; No allocation needed - - # foo - ; Argument handling - check_argcount 0 0 ; ip == 3 - - ; FCall arguments - push_literal #<"foo"> ; ip == 5 - string_dup ; ip == 6 - - ; FCall receiver - push_self ; ip == 7 - - ; FCall send - set_call_flags 1 ; ip == 9 - meta_send_stack_1 :puts ; ip == 11 - - ; Cleanup - sret ; ip == 12 + # Toplevel + ; No allocation needed + + # foo + ; Argument handling + check_argcount 0 0 ; ip == 3 + + ; FCall arguments + push_literal #<"foo"> ; ip == 5 + string_dup ; ip == 6 + + ; FCall receiver + push_self ; ip == 7 + + ; FCall send + set_call_flags 1 ; ip == 9 + meta_send_stack_1 :puts ; ip == 11 + + ; Cleanup + sret ; ip == 12 -Code generation responsibility shifts again to the original `MethodDescription` -and its `$gen`. +Code generation responsibility shifts again to the original MethodDescription +and its $gen. -`#compile_body` actually returns the new `MethodDescription` we have +#compile_body actually returns the new MethodDescription we have been working with so we can finally store the literal and then have -the newly generated method added to `self` which in our case is the -top-level entity. Notably, the `:add_method` operation handles looking -up its argument (the name) in literals before generating the real +the newly generated method added to self which in our case is the +top-level entity. Notably, the +:add_method+ operation handles looking +up its argument (the name) in literals before generating the real instruction so it is not necessary to do that part. - # Toplevel - ; Add method - push_literal #<foo> ; ip == 2 - push_self ; ip == 3 - add_method :foo ; ip == 5 - - # foo - ; Argument handling - check_argcount 0 0 ; ip == 3 - - ; FCall arguments - push_literal #<"foo"> ; ip == 5 - string_dup ; ip == 6 - - ; FCall receiver - push_self ; ip == 7 - - ; FCall send - set_call_flags 1 ; ip == 9 - meta_send_stack_1 :puts ; ip == 11 - - ; Cleanup - sret ; ip == 12 - - - Script - .body = Newline - .file = 'test.rb' - .line = 1 - .child = Define - ... - -Nothing to do in the next `Newline` either so eventually we just end -up at the very topmost `Script#bytecode` which merely `#pop`s whatever + # Toplevel + ; Add method + push_literal #<foo> ; ip == 2 + push_self ; ip == 3 + add_method :foo ; ip == 5 + + # foo + ; Argument handling + check_argcount 0 0 ; ip == 3 + + ; FCall arguments + push_literal #<"foo"> ; ip == 5 + string_dup ; ip == 6 + + ; FCall receiver + push_self ; ip == 7 + + ; FCall send + set_call_flags 1 ; ip == 9 + meta_send_stack_1 :puts ; ip == 11 + + ; Cleanup + sret ; ip == 12 + + Script + .body = Newline + .file = 'test.rb' + .line = 1 + .child = Define + ... + +Nothing to do in the next Newline either so eventually we just end +up at the very topmost Script#bytecode which merely #pops whatever the return value of the method definition is (since there is no use -for it and it cannot be left to litter the stack), `#push`es true +for it and it cannot be left to litter the stack), #pushes true as its own "return value" and returns. - # Toplevel - ; Add method - push_literal #<foo> ; ip == 2 - push_self ; ip == 3 - add_method #<:foo> ; ip == 5 - + # Toplevel + ; Add method + push_literal #<foo> ; ip == 2 + push_self ; ip == 3 + add_method #<:foo> ; ip == 5 + + ; Cleanup + pop ; ip == 6 + push_true ; ip == 7 + sret ; ip == 8 + + # foo + ; Argument handling + check_argcount 0 0 ; ip == 3 + + ; FCall arguments + push_literal #<"foo"> ; ip == 5 + string_dup ; ip == 6 + + ; FCall receiver + push_self ; ip == 7 + + ; FCall send + set_call_flags 1 ; ip == 9 + meta_send_stack_1 :puts ; ip == 11 + ; Cleanup - pop ; ip == 6 - push_true ; ip == 7 - sret ; ip == 8 - - # foo - ; Argument handling - check_argcount 0 0 ; ip == 3 - - ; FCall arguments - push_literal #<"foo"> ; ip == 5 - string_dup ; ip == 6 + sret ; ip == 12 - ; FCall receiver - push_self ; ip == 7 - - ; FCall send - set_call_flags 1 ; ip == 9 - meta_send_stack_1 :puts ; ip == 11 - - ; Cleanup - sret ; ip == 12 - -Execution takes us back to the first `MethodDescription#run` which has -one last thing to do: `#$gen.close`. We now know that line 1 ends at -`$gen` ip == 8 (and started at ip 2.) At this point, we return to caller. -Now we can actually create the `CompiledMethod`. +Execution takes us back to the first MethodDescription#run which has +one last thing to do: #$gen.close. We now know that line 1 ends at +$gen ip == 8 (and started at ip 2.) At this point, we return to caller. +Now we can actually create the CompiledMethod. If you want to know what the instructions above do, you can look at the -"opcode docs":http://rubini.us/doc/vm. +{opcode docs}[http://rubini.us/doc/vm]. - desc = ast.to_description - compiled = desc.to_cmethod # -> $gen.to_cmethod desc + desc = ast.to_description + compiled = desc.to_cmethod # -> $gen.to_cmethod desc - -Stage 3: Encoding and CompiledMethod -==================================== +== Stage 3: Encoding and CompiledMethod One clarification: Rubinius does not actually use BYTEcode in the sense of one-byte-wide opcodes so it is used here in its banal meaning of virtual -machine machine code. Rubinius' machine code is in fact INTcode. `encoder.rb` -will be the main venue here (although, again, "the VM docs":http://rubini.us/doc/vm -are an excellent resource for anything to do with Rubinius bytecode.) - -The process of creating a `CompiledMethod` starts from `Generator#to_cmethod` -to which we get through `MethodDescription#to_cmethod`. The first thing that -happens is that the `Generator` goes through its entire stream, "collapsing" -any `Label`s it encounters. You can review (TODO: link to label text) for -more info but suffice to say `Labels` are just positions is the stream that -can be used for branching, gotos and jumps. Our code has no `Label`s so we -need not worry about it. The first thing we need to do is have the `Encoder` -convert our "assembly code" into an `InstructionSequence`. - -If you look at `encoder.rb`, it is almost deceptively simple. At the top there +machine machine code. Rubinius' machine code is in fact INTcode. encoder.rb +will be the main venue here (although, again, {the VM +docs}[http://rubini.us/doc/vm] are an excellent resource for anything to do +with Rubinius bytecode.) + +The process of creating a CompiledMethod starts from Generator#to_cmethod +to which we get through MethodDescription#to_cmethod. The first thing that +happens is that the Generator goes through its entire stream, "collapsing" +any Labels it encounters. You can review (TODO: link to label text) for +more info but suffice to say Labels are just positions is the stream that +can be used for branching, gotos and jumps. Our code has no Labels so we +need not worry about it. The first thing we need to do is have the Encoder +convert our "assembly code" into an InstructionSequence. + +If you look at encoder.rb, it is almost deceptively simple. At the top there are several data structures to help us keep track of the various opcodes and any special properties they may have. Towards the bottom, there is only a little tiny bit of code that is supposed to handle everything for us. -We start off with `Encoder#encode_stream`, expecting back an `InstructionSequence.` -`#encode_stream` first calculates the size required for the iseq and then creates -one. Then it simply `#encode`s each operation in the stream, keeping track of -the offset into the `InstructionSequence`. +We start off with Encoder#encode_stream, expecting back an InstructionSequence. +#encode_stream first calculates the size required for the iseq and then creates +one. Then it simply #encodes each operation in the stream, keeping track of +the offset into the InstructionSequence. -`#encode` is not a very complicated method, either: the numeric opcode is -looked up and then its "width" or number of additional arguments is -determined from the lookup tables at the top of `encoder.rb`. For each +#encode is not a very complicated method, either: the numeric opcode is +looked up and then its "width" or number of additional arguments is +determined from the lookup tables at the top of encoder.rb. For each one of these elements (maximum three, opcode + 0, 1 or 2 arguments) gets -encoded into the `InstructionSequence` using `Encoder#int2str`. +encoded into the InstructionSequence using Encoder#int2str. -`#int2str` just writes the given number into the iseq as a big-endian 4-byte +#int2str just writes the given number into the iseq as a big-endian 4-byte integer split into 4 byte-sized segments. For example, 17 would end up as -`[0, 0, 0, 17]` whereas 256 becomes `[0, 0, 1, 0]` and so on. The fully -encoded `InstructionSequence` is then returned to `Generator#to_cmethod`. - -Next we create the `CompiledMethod` object (which, by the way, is immensely -cool, do `var = def my_method ...` in `sirb` and see what all you can do -with `var`! Also see TODO: Rubinius reflection capabilities) using the -`InstructionSequence` just created. The `CM` is not quite ready yet, though, -so we store its `name`, the `file` it was defined in (these are either real -names or one of the special names such as `:__eval__` or `:__unknown__`) as +[0, 0, 0, 17] whereas 256 becomes [0, 0, 1, 0] and so on. The fully +encoded InstructionSequence is then returned to Generator#to_cmethod. + +Next we create the CompiledMethod object (which, by the way, is immensely +cool, do var = def my_method ... in sirb and see what all you can do +with var! Also see TODO: Rubinius reflection capabilities) using the +InstructionSequence just created. The CM is not quite ready yet, though, +so we store its name, the file it was defined in (these are either real +names or one of the special names such as +:__eval__+ or +:__unknown__+) as well as the serial which is essentially a version number. -Before you think I forgot about `foo`, that comes next as `#to_cmethod` -goes through `#encode_literals`, `#encode_lines` and `#encode_exceptions` -(as you rememeber, the `MethodDescription` for `foo` was stored as a literal.) +Before you think I forgot about foo, that comes next as #to_cmethod +goes through #encode_literals, #encode_lines and #encode_exceptions +(as you rememeber, the MethodDescription for foo was stored as a literal.) All three of those are very similar and do not really perform any actual -encoding like we saw before. Each just goes through its ascribed list -and stores the objects in a `Tuple` (a very primitive fixed-size container) -and the only exception to this is `#encode_literals` and then only for -any `MethodDescription`s it finds. For those it calls `lit.to_cmethod` -which obviously then recursively generates the `CompiledMethods`. The +encoding like we saw before. Each just goes through its ascribed list +and stores the objects in a Tuple (a very primitive fixed-size container) +and the only exception to this is #encode_literals and then only for +any MethodDescriptions it finds. For those it calls lit.to_cmethod +which obviously then recursively generates the CompiledMethods. The generated ones get stored in the literals tuple in place of the old -`MethodDescription`. +MethodDescription. -And that is all there is to it. The `CompiledMethod` is finished and -it can be stashed away somewhere (a `MethodTable`, for example), it -can be `#activate`d to have the code run and so on. If you are interested +And that is all there is to it. The CompiledMethod is finished and +it can be stashed away somewhere (a MethodTable, for example), it +can be #activated to have the code run and so on. If you are interested in what happens when one IS activated, read TODO: link to execution. -Well, maybe there is one little thing that might deserve to be -covered still: how can `CompiledMethod`s be stored on disk in `.rbc` -files? - - +Well, maybe there is one little thing that might deserve to be +covered still: how can CompiledMethods be stored on disk in .rbc +files? -Stage 4: Bytecode on disk. -=========================== +== Stage 4: Bytecode on disk. -The loading and unloading of code -- `CompiledMethod`s -- to disk -happens through `Marshal` and is mostly defined in `shotgun/lib/cpu_marshal.c`. +The loading and unloading of code -- CompiledMethods -- to disk +happens through Marshal and is mostly defined in shotgun/lib/cpu_marshal.c. -The entire thing, from a high level, is fairly trivial. `cpu_marshal()` first -brands the file with the special string `"RBIX"` followed by a version number +The entire thing, from a high level, is fairly trivial. cpu_marshal() first +brands the file with the special string "RBIX" followed by a version number to be able to detect the formatting required (when loading back again.) From -there, `cpu_marshal_cmethod2()` takes care of the rest. The basic idea for +there, cpu_marshal_cmethod2() takes care of the rest. The basic idea for marshalling is simple although it varies slightly from one object to another: -first a type identifier is printed ("M" for a `CompiledMethod`) possibly -followed by some type of version identifier or a size (a string, for +first a type identifier is printed ("M" for a CompiledMethod) possibly +followed by some type of version identifier or a size (a string, for example, will write the number of characters) that will allow unmarshalling the object later. After this comes the object data. -`cpu_marshal_cmethod2()` loops through the 16 fields of a `CompiledMethod` -and recursively marshals them. For example, the name (field 5) is a `Symbol` -which uses the format `'x' + size + characters` so for our method from -above, the end-result would be, in hex: +cpu_marshal_cmethod2() loops through the 16 fields of a CompiledMethod +and recursively marshals them. For example, the name (field 5) is a Symbol +which uses the format 'x' + size + characters so for our method from +above, the end-result would be, in hex: - 78 00 00 00 03 66 6f 6f - x 3 f o o + 78 00 00 00 03 66 6f 6f + x 3 f o o -Unmarshalling back into a `CompiledMethod` is a very similar process. -All that happens is that a new, blank `CompiledMethod` is allocated +Unmarshalling back into a CompiledMethod is a very similar process. +All that happens is that a new, blank CompiledMethod is allocated and then its fields are unmarshalled using a similar process. The -end-result is a finished `CompiledMethod`. +end-result is a finished CompiledMethod. Easy, right? The implementation, of course, is a bit more involved (mostly because of working with files) but from our perspective it -is child's play. Do rememeber, though, that this marshal format is -*not* compatible with MRI, not even for the parts that exist in -both implementations such as `String`s. +is child's play. Do rememeber, though, that this marshal format is +*not* compatible with MRI, not even for the parts that exist in +both implementations such as Strings. doc/compiler_intro.txt @@ -1,26 +1,26 @@ Control modification in CRuby. Behavior of control modification keywords on blocks. - All take an argument, nil by default. -next: soft return from the block, the blocks return value - is the argument to next. +All take an argument, nil by default. -break: hard return from the method that called the block. - the methods return value is the argument to break. +[next] soft return from the block, the blocks return value + is the argument to next. -redo: the ip is reset to the start of the block and - execution is continued. +[break] hard return from the method that called the block. + the methods return value is the argument to break. -retry: the ip is reset to the start of the method calling - the block and the method is completely rerun, including - reimporting the arguments into locals. +[redo] the ip is reset to the start of the block and execution is continued. + +[retry] the ip is reset to the start of the method calling + the block and the method is completely rerun, including + reimporting the arguments into locals. I question the fact that the argument reimporting code is run on a retry. You can't have a bare retry in a method, only inside a block will it work. That being true, I also question retry in a block that influences the calling method directly -in the first place. +in the first place. The one good aspect of having some behavior for retry is that it can't be used as the retry for a begin/rescue/end. This @@ -29,31 +29,31 @@ the sending context of a block. This means that conditional modifiers can be optimized directly into bytecode gotos. Exceptions to this rule in CRuby. -Proc's: no retry available. Because Proc's are created by calling +Proc: no retry available. Because Proc's are created by calling proc() or Proc.new with a block, returning INTO proc() or Proc.new makes no sense. Thread: all control mods just exit the thread. -==================== +--- For while and until. -next: return to the top of the conditional. -break: jump to just after the whiles body. -redo: return to the top of the while, after the conditional though. -retry: NOT ALLOWED +[next] return to the top of the conditional. +[break] jump to just after the whiles body. +[redo] return to the top of the while, after the conditional though. +[retry] NOT ALLOWED No mods for if, unless, and case. For the body of a begin: -next, redo, retry: NA -break: jump to just beyond the end of the begin. +[next, redo, retry] NA +[break] jump to just beyond the end of the begin. Inside a rescue: -next, break, redo: NA -retry: Jump back to the top of the begin. +[next, break, redo] NA +[retry] Jump back to the top of the begin. Inside an ensure/else: -next, break, redo, retry: NA +[next, break, redo, retry] NA doc/cond_mod.txt @@ -1,20 +1,19 @@ -How exceptions are delivered. += How exceptions are delivered. -There are 3 separate parts: -1) The exception deliver table, one per compiled method. -2) A translator that converts rescues into the exception table -3) The exception deliverer, which searches tables and perform jumps. +There are 3 separate parts +1. The exception deliver table, one per compiled method. +2. A translator that converts rescues into the exception table +3. The exception deliverer, which searches tables and perform jumps. - -The exception table. +== The exception table. This is the core of the exception deliver mechanism. The table is organized into rows, each row having 3 columns: - 1) the start bytecode index - 2) the end bytecode index - 3) the bytecode index to jump to +1. the start bytecode index +2. the end bytecode index +3. the bytecode index to jump to -How they all work together: +== How they all work together When an exception is raised, the deliverer takes the current context's exception table and the current ip and searches @@ -43,22 +42,20 @@ find someone to handle the exception, the cpu is trigger to handle it itself. This typically means informing the user in some way that an exception has reached the top and that the program is going to terminate. -Algorithms: +== Algorithms find an exception handler: -I = @active_context.ip -T = @active_context.exceptions - -while @active_context - - T.each_row do |entry| - if entry[0] <= I and entry[1] >= I - @cpu.ip = entry[2] - return - end - end - - @active_context = @active_context.sender - -end \ No newline at end of file + I = @active_context.ip + T = @active_context.exceptions + + while @active_context + T.each_row do |entry| + if entry[0] <= I and entry[1] >= I + @cpu.ip = entry[2] + return + end + end + + @active_context = @active_context.sender + end doc/exceptions.txt @@ -1,18 +1,15 @@ -The life of a context. += The life of a context. -============ - -Definitions: +== Definitions: Virtual bottom (om->context_bottom): The lowest context, address wise, in the stack that is not referenced. -=========== - -Description: +== Description: All contexts begin life in the context stack (state->om->contexts). -A context is allocated at the next available position in the stack (contexts->current). +A context is allocated at the next available position in the stack +(contexts->current). The context is then activated. When another method call is performed, the cpu registers are saved into it, and it is registered as the new context's sender. @@ -24,21 +21,22 @@ Now there are 3 paths: context is always on the bottom. So the space it took up is decremented from contexts->current, so the next context uses the same memory. -2. Same as 1, but the context was referenced. Before decrementing contexts->current, - the address of the context is compared against the virtual bottom of the - stack. Because the context has been referenced, it is equal to the virtual - bottom, so the context stack is not decremented, leaving this context alive - in the context stack. +2. Same as 1, but the context was referenced. Before decrementing + contexts->current, the address of the context is compared against the + virtual bottom of the stack. Because the context has been referenced, it is + equal to the virtual bottom, so the context stack is not decremented, + leaving this context alive in the context stack. 3. Same as 2, but instead a context created by a method send in this context is referenced. When this context goes to return, it's address is lower than the virtual bottom, so the contexts->current is left unchanged. -Phase 2 +== Phase 2 When the baker GC is run, the first thing it does is formalize all contexts between the true bottom and the virtual bottom. These are contexts that ones -that were directly referenced, or indirectly referenced (a child was referenced). +that were directly referenced, or indirectly referenced (a child was +referenced). The baker GC is then run. Any references to contexts in the context stack will be of contexts between the true bottom and the virtual bottom only, and when @@ -56,73 +54,67 @@ the rest of the contexts still in the context stack, moving them one after another closer to the bottom. This compacts the context stack, removing all contexts that have been placed into the baker heap. -Phase 2.5 +== Phase 2.5 When the mark/sweep GC is run, the contexts on the context stack are walked specifically, marking the objects that they point to. This is the only extra effort that the mark/sweep GC requires of the context stack. -Phase 3: Baker GC context death +== Phase 3: Baker GC context death When a context store in the baker GC returns, it's sender is set to nil (to break the chain) and it is left in the heap, to be cleaned up the next time the GC runs. - -============================= - -heap == baker GC heap +== heap == baker GC heap Scenarios: 0. no contexts - -A: nil -B: on the stack - -B->sender is nil, and B will be on the stack, as it's the first context. - + A. nil + B. on the stack + + B->sender is nil, and B will be on the stack, as it's the first context. 1. all on stack. - -A: on stack -B: child of A, on stack - -on GC, B->sender is adjusted for new location of A, post compacting. -if A has not been referenced, A is the ctx located in memory just before B. - + A. on stack + B. child of A, on stack + + on GC, B->sender is adjusted for new location of A, post compacting. + if A has not been referenced, A is the ctx located in memory just before B. 2. heap and stack. - -A: in heap -B: child, on stack - -on GC, B->sender is compacted by baker, returning new location. -B is located at the virtual bottom. - + A. in heap + B. child, on stack + + on GC, B->sender is compacted by baker, returning new location. + B is located at the virtual bottom. 3. both in heap - -A: in heap -B: child, in heap - -on GC, B->sender is compacted by baker, returning new location. - + A. in heap + B. child, in heap + + on GC, B->sender is compacted by baker, returning new location. 4. stack and heap + A. on stack + B. child, in heap (can only be BlockContext) + + A is referenced, thus below virtual bottom. On GC of B, B->sender causes A + to be copied into heap, returning new location. -A: on stack -B: child, in heap (can only be BlockContext) +== Assertions: -A is referenced, thus below virtual bottom. On GC of B, B->sender causes A to be -copied into heap, returning new location. +A context can never be in the mark_sweep GC. +A context referenced in the heap must be also in heap, or below virtual bottom +(about to be copied) -======== +A sender on the stack must be in the heap or exactly CTX_SIZE less that the +current context. -Asssertions: +Only the context closest to the virtual bottom may be referenced by an object +in the heap. -A context can never be in the mark_sweep GC. -A context referenced in the heap must be also in heap, or below virtual bottom (about to be copied) -A sender on the stack must be in the heap or exactly CTX_SIZE less that the current context. -Only the context closest to the virtual bottom may be referenced by an object in the heap. If a context is on the stack and not referenced, it's class is nil. + If a context is on the stack and referenced, it's class is fastctx. -If a context is in the heap, it's class is either fastctx or blokctx. + +If a context is in the heap, it's class is either fastctx or blokctx. doc/life_of_a_context.txt @@ -1,39 +1,36 @@ -== Method Contract == += Method Contract Methods contain a contract that method implements and the caller must adhere to. Herein are pieces of the method contract. -A) The method must be called with the correct number of arguments. - 1) If a method provides a fixed number of arguments, the +A. The method must be called with the correct number of arguments. + 1. If a method provides a fixed number of arguments, the caller must provide exactly that number of arguments. - 2) If a method is varadic (contains a splat), then the caller + 2. If a method is varadic (contains a splat), then the caller must provide at least the number of non-varadic arguments. - 3) If a method contains defaults for arguments, the default values + 3. If a method contains defaults for arguments, the default values will be assigned to the local variable if there is no assignment by the caller. - 4) If a method is called with the incorrect number of arguments, + 4. If a method is called with the incorrect number of arguments, it raises an ArgumentError to indicate the error to the caller. - -B) If the method captures the block as a local variable, it must be +B. If the method captures the block as a local variable, it must be assigned before the body of the method begins. +C. As hinted to in A2, a method may provide default values for arguments. + The default values are evaluated in the context of the method, not the + caller. -C) As hinted to in A2, a method may provide default values for arguments. - 1) The default values are evaluated in the context of the method, - not the caller. - -=== Future Contracts === += Future Contracts -D) The method provides names for each argument. - 1) The caller may provide values for those arguments by name. - 2) Any arguments provided by name must be provide after all +A. The method provides names for each argument. + 1. The caller may provide values for those arguments by name. + 2. Any arguments provided by name must be provide after all arguments provided by position only. - 3) There must not be a positional value and keyword value + 3. There must not be a positional value and keyword value for an argument. If there is 2 values for a single argument, a ArgumentError exception is thrown. - 4) If the method specifies a varadic (splat) argument, unused + 4. If the method specifies a varadic (splat) argument, unused keywords will be stored in the splat (see E for details). - -E) Where D is true, the splat is not a simple Array, but rather a +B. Where D is true, the splat is not a simple Array, but rather a MethodArguments (MA) object. MA has a to_ary method that will return the normal splat. MA also provides methods - for accessing unused keywords. \ No newline at end of file + for accessing unused keywords. doc/methods.txt @@ -1,10 +1,10 @@ The point of the bootstrap step is to build up just enough functionality to: -1) use alias -2) run FFI (for platform code) -3) provide basic error reporting (primary related to method_missing) +1. use alias +2. run FFI (for platform code) +3. provide basic error reporting (primary related to method_missing) -It also serves to attach all primitives. Primitives are ONLY allowed to be +It also serves to attach all primitives. Primitives are *only* allowed to be attached in bootstrap. All functionality provided by bootstrap may be redefined in core to provide @@ -12,5 +12,5 @@ a 'proper' implementation. Rules of bootstrap: -1) limited to no metaprogramming. The bootstrap should be a container of code only. - if it runs code, then order starts to matter and things fall apart. +1. Limited to no metaprogramming. The bootstrap should be a container of code + only. If it runs code, then order starts to matter and things fall apart. doc/point_of_bootstrap.txt 11c51bdad0806a07f239bf44a4e45c96dd3233b4 Eric Hodel ehodel@engineyard.com http://github.com/evanphx/rubinius/commit/edc440c85ec5d2c895daf8b83923798422d20797 edc440c85ec5d2c895daf8b83923798422d20797 2008-02-12T21:01:36-08:00 2008-02-12T21:01:36-08:00 RDocify (but not hooked into anything yet). 32b76b2a40ae4d8a70d9f76479672c7492c09161 Eric Hodel ehodel@engineyard.com