Syntax Grammar

The MRI syntax grammar doesn't lend itself to precise formal documentation in the typical fashion for programming languages, because the parsing process is implemented in a fairly complicated way that can occasionally be inconsistent and/or buggy, and to fully describe what the permitted syntax is would require a myriad of detailed rules for exceptional circumstances that wouldn't be much use to anyone.

However, this is not a problem, because we can easily describe the syntax of Ruby by showing all the important elements, but point out where odd, but harmless, exceptional behaviours can occur, and leave these behaviours undefined. This way gives the intent of the syntax grammar without getting bogged down in implementation limitations.

I've described the grammar of specific syntax elements in other sections - this page explains the rules for combining expressions together.

#Statements

A statement is any Ruby expression terminated either by ; or a newline character, within the context of a block of statements. A block of statements is executed iteratively in the typical fashion. A block of statements is permitted as part of various other expressions e.g. def, class, module, if, rescue etc.

A block of statements can be nested within another Ruby expression by enclosing it within ( and ), or by using a begin/end expression (i.e. enclosing between begin and end, note that begin/end expressions also have some other specific behaviours). The return value of a block of statements is the return value of the last statement in the block.

A Ruby sourcefile is a block of statements.

##Newline terminators Statements can be made to run across multiple lines of code for code readability purposes - if a newline character occurs mid-statement and it can't possibly be the end of a statement, Ruby will often ignore the newline.

However, the Ruby implementation of this is fairly simplistic, based on what the last token before the newline was and without taking into account the broader context of how the newline is nested and what's on the following line. Therefore, there are scenarios where Ruby will not ignore the newline even though it probably should. For example, Ruby won't necessarily know to ignore a newline in the following places:

• In the middle of an argument list which is inside parentheses/brackets/braces, unless the line ends in , or some other token that makes it obvious the statement has not finished (e.g. operators, opening parentheses/brackets/braces)
• Exception: Ruby will generally ignore a newline that occurs just before closing parentheses/brackets/braces
• If the code on the next line can't possibly be a new statement or other valid syntax (and so must be part of the current line's statement)
• Exception: if the next line begins (after whitespace) with a ., Ruby will ignore the newline, so you can chain method calls and insert a newline before the .
• Exception: In the ternary ?: operator, Ruby will ignore a newline directly before the :

#Control transfer expression restrictions Ruby will generally raise a compile error if you use a control transfer expression (return/break/next/redo/retry) in a place which requires a value and it doesn't really make sense to transfer control (e.g. assignment expressions, operator expressions, conditional expressions, call arguments, explicitly scoped method calls/constant references). The main exception is that you can use a control transfer instruction on the right hand side of logical and/or operators, in order to take advantage of the short-circuiting feature.

#Hierarchy of expressions In general I've simplified and said that any Ruby expression can be used in any part of Ruby code, but there is actually a hierarchy of expressions, with each step more restrictive than the next, and different parts of Ruby code permit expressions from different levels of the hierarchy. However, this doesn't ultimately restrict the expressiveness of the language because you can always use any expression anywhere if it is enclosed in ( and ).

The hierarchy of expressions restricts what types of expressions are permitted in a given place in Ruby code. It also generally defines a sort of precedence for expressions - because expressions typically have sub-expressions permitted from the same or a more restricted level of the hierarchy, more restricted levels take precedence over less restricted levels. Technically, operator precedence (as described in the Operators section) usually only applies to determine precedence between operator expressions on the same level of the hierarchy, however the operator precedence table I've provided is effective at describing precedence relationships between different levels of the hierarchy as well.

The 4 levels of the hierarchy (from least to most restrictive) are:

• Statement grammar
• Condition grammar
• Argument grammar
• Primary grammar

##Statement grammar The statement grammar contains expressions that are not permitted to be nested anywhere, because they have unusual syntax structures or interpretations, or have potential to cause syntax ambiguities if nested within other expressions.

Statement grammar expressions are used inside:

• Blocks of statements
• The right-hand-side of a rescue statement modifier (but not an assignment modifier)

Permitted expressions in statement grammar are:

• All condition grammar expressions
• alias/undef/BEGIN{}/END{} expressions
• Assignments where the right-hand-side is a command call (including chained simple assignments i.e. a = b = method arg)
• Assignments with a multiple left- or right-hand-side
• Statement modifiers (if/unless/while/until/rescue)

##Condition grammar The condition grammar is where the command call lives and is used in places where command calls are permitted (generally because it's not the middle of an argument list and so there won't be confusion about commas separating arguments). This is also where the and, or and not operators live, whose most notable characteristic is that they can have command calls as their operands (unlike &&, || and !). Therefore this level of the hierarchy has the full suite of logical operations for conditional expressions.

Condition grammar expressions are used inside:

• The condition expression in if/unless/while/until expressions
• The value to be matched in case expressions
• The value to be iterated in for expressions
• The operands of and, or, not
• The superclass in class expressions
• The object whose singleton class is targeted in class/def expressions
• The operand of defined? if enclosed in parentheses and there is no whitespace preceding it

Permitted expressions in condition grammar are:

• All argument grammar expressions
• Command calls (including command call forms of super/yield/break/next/return)
• and/or/not operators
• ! <command_call> (as a special concession for the ! operator which normally can't take a condition grammar operand)

##Argument grammar Argument grammar is the most common hierarchical level for nested expressions, it's used almost everywhere including call arguments and operator operands. It contains all expressions that can be nested within a comma separated list (e.g. call arguments), including all operators and assignments.

Argument grammar expressions are used inside:

• Call arguments and similar (including array references, array literals, hash literals, when clauses, rescue clauses and multiple right-hand-side assignments)
• Operands for all operators, including assignment right-hand-sides, but excluding and, or, not and the condition grammar form of defined?
• Optional argument default values (except for blocks, whose default values are primary grammar so as not to conflict with the enclosing |)
• The rescue assignment modifier's value

Permitted expressions in argument grammar are:

• All primary grammar expressions
• All operators excluding and, or, not
• All assignment operators (with argument grammar right-hand-sides)

##Primary grammar Primary grammar is that which you can invoke a method/reference a constant on by way of ., :: or [] as an array reference. These syntax elements essentially have the highest precedence (except for special uses of defined? and not) and so the primary grammar contains mostly method/constant references and expressions that are unmistakably a single unit for the method call/constant reference to operate on.

Primary grammar expressions are used inside:

• The left side of a . or :: method call/constant reference
• The array value in an array reference
• The value of optional arguments in a block

Permitted expressions in primary grammar are:

• "Single unit" expressions, either a single token or encapsulated tokens:
• Literals
• Variable references
• Lambda expressions
• A block of statements between ( and )
  • Note that when this occurs as the first argument of a command call, for some reason I can't fathom its content must be a condition expression, not a block of statements
• All the remaining "statement-like" expressions except for method-call-like expressions (if, unless, while, until, case, for, begin/end, class, module, def, redo, retry)
• Method/constant references (which can be chained from left to right)
• Method calls (including super/yield/break/next/return, but excluding command calls)
• Constant references (including explicitly namespaced)
• Highest priority operators
• defined? and not if their argument is enclosed in parentheses and there is no whitespace between the operator and the parentheses

Command calls

Command calls are method calls or super/yield/break/next/return calls with at least one argument, where the argument list is not encapsulated in parentheses.

Command calls are permitted:

• In the condition grammar
• In assignment expressions in the statement grammar
• As the single and only call argument to a method call (including array references)

In the presence of command calls, there is a difference between blocks enclosed in { and } and those enclosed in do and end - brace blocks will apply to the innermost nested method call (command call or otherwise), and do blocks will apply to the outermost command call. This applies when the last argument to your command call is a method call (e.g. cmd1 method1(arg1) { ... } or cmd1 cmd2 arg1 { ... }).

There are a number of implementation limitations regarding blocks on command calls. One is that you may not put a do ... end block on a command call nested inside a method call (e.g. method(cmd arg1 do ... end) fails). Also, there are only a few situations where it works to put a brace block on a command call, and the implementation is buggy and will sometimes give a compile time error incorrectly. If you want to use a brace block after a command call, try putting one directly after a ] or }, but it's probably better to make it a non-command-style method call.

Command call chaining

Regular method calls can be chained because they are in the primary grammar.

call1(arg1, arg2).call2

A command call couldn't generally be chained to another method call because the method call would apply to the command call's final argument, instead of to the command call.

call1 arg1, arg2.call2

However, a command call with a block is a different story - there would be no conflict regarding what the chained call applies to, it must apply to the command call.

call1 arg1, arg2 { blockcode }.call2

Conceptually, a command call with a block like this could almost be part of the primary grammar, and be chainable to other calls with ., :: and [] as well as being the left-hand-side of binary operators. However, it isn't, but Ruby does provide limited support for command call chaining. A command call can be chained as follows:

• It must end with a do ... end block (not a brace block)
• This may be followed by 0 or more chained non-command-call-style method calls (the arguments, if any, are in parentheses) and which have no blocks
• This may be optionally be followed by one chained command-call-style method call (the arguments are not in parentheses) which has no block

Conflict resolution

In certain circumstances, there can be multiple interpretations of code. This is a conflict of syntax interpretation, and the rules for resolving these conflicts are as follows.

Brace blocks

From highest to lowest precedence:

1. Lambdas - e.g. -> myfunc arg=gets() { ... } the brace block is the lambda body, not a block attached to the gets() call
2. Innermost method/command call - e.g. cmd1 method1(arg1) { ... } the brace block attaches to the method1 call, not the cmd1 call.
3. Hash constructors - brace-encapsulated code will only be considered a hash constructor if it's not possible, based on its context, for it to be a brace block

Do blocks

From highest to lowest precedence:

1. Lambdas - e.g. -> arg=gets() do ... end the do block is the lambda body, not a block attached to the gets() call
2. while/until/for statements - e.g. while gets() do ... end the do block is the loop body, not a block attached to the gets() call
3. Outermost command call - e.g. cmd1 method1(arg1) do ... end the do block attaches to the cmd1 call, not the method1 call.

Command call first argument

There are many syntax elements that could be interpreted in 2 different ways, if it's possible they are a command call's first argument. For example, a * b could be interpreted as (a) * (b), i.e. multiplication, or as a (*b), i.e. a command call with a splat argument.

In general, the conflict resolution rules are as follows:

• If the conflicting syntax element is preceded by whitespace, but not followed by it - it's a command call argument
• Otherwise, it's not a command call argument
• As a special exception to these rules - the conflicting syntax element will never be judged a command call argument if it would be an implicitly targeted command call on a method with the same name as a local variable in scope. For example, method *arg would normally be a command call, but if there is a local variable in scope named method, this is a multiplication.

The table of conflicting syntax elements, and their different interpretations, follows:

Token	Regular meaning	Command call argument meaning	Notes
*	Binary multiply	Splat argument
&	Binary bitwise and	Block argument
+	Binary plus	Unary plus
-	Binary minus	Unary minus
/	Binary divide	Start of regular expression literal
%	Binary modulus	Start of %... literal
<<	Binary left shift	Heredoc
?	Ternary ?:	Start of character literal
::	Binary reference	Unary reference (on Object)
:	Ternary ?:	Start of symbol literal

Character processing details - newlines, eof reserved words