Syntax Grammar

The MRI syntax grammar doesn't lend itself to precise formal documentation in the typical fashion for programming languages, because the parsing process is implemented in a fairly complicated way that can occasionally be inconsistent and/or buggy, and to fully describe what the permitted syntax is would require a myriad of detailed rules for exceptional circumstances that wouldn't be much use to anyone.

However, this is not a problem, because we can easily describe the syntax of Ruby by showing all the important elements, but point out where odd, but harmless, exceptional behaviours can occur, and leave these behaviours undefined. This way gives the intent of the syntax grammar without getting bogged down in implementation limitations.

I've described the grammar of specific syntax elements in other sections - this page explains the rules for combining expressions together.

#Statements

A statement is any Ruby expression terminated either by ; or a newline character, within the context of a block of statements. A block of statements is executed iteratively in the typical fashion. A block of statements is permitted as part of various other expressions e.g. def, class, module, if, rescue etc.

A block of statements can be nested within another Ruby expression by enclosing it within ( and ), or by using a begin/end expression (i.e. enclosing between begin and end, note that begin/end expressions also have some other specific behaviours). The return value of a block of statements is the return value of the last statement in the block.

A Ruby sourcefile is a block of statements.

##Newline terminators Statements can be made to run across multiple lines of code for code readability purposes - if a newline character occurs mid-statement and it can't possibly be the end of a statement, Ruby will often ignore the newline.

However, the Ruby implementation of this is fairly simplistic, based on what the last token before the newline was and without taking into account the broader context of how the newline is nested and what's on the following line. Therefore, there are scenarios where Ruby will not ignore the newline even though it probably should. For example, Ruby won't necessarily know to ignore a newline in the following places:

• In the middle of an argument list which is inside parentheses/brackets/braces, unless the line ends in , or some other token that makes it obvious the statement has not finished (e.g. operators, opening parentheses/brackets/braces)
• Exception: Ruby will generally ignore a newline that occurs just before closing parentheses/brackets/braces
• If the code on the next line can't possibly be a new statement or other valid syntax (and so must be part of the current line's statement)
• Exception: if the next line begins (after whitespace) with a ., Ruby will ignore the newline, so you can chain method calls and insert a newline before the .
• Exception: In the ternary ?: operator, Ruby will ignore a newline directly before the :

#Control transfer expression restrictions Ruby will generally raise a compile error if you use a control transfer expression (return/break/next/redo/retry) in a place which requires a value and it doesn't really make sense to transfer control (e.g. assignment expressions, operator expressions, conditional expressions, call arguments, explicitly scoped method calls/constant references). The main exception is that you can use a control transfer instruction on the right hand side of logical and/or operators, in order to take advantage of the short-circuiting feature.

#Hierarchy of expressions In general I've simplified and said that any Ruby expression can be used in any part of Ruby code, but there is actually a hierarchy of expressions, with each step more restrictive than the next, and different parts of Ruby code permit expressions from different levels of the hierarchy. However, this doesn't ultimately restrict the expressiveness of the language because you can always use any expression anywhere if it is enclosed in ( and ).

The hierarchy of expressions restricts what types of expressions are permitted in a given place in Ruby code. It also generally defines a sort of precedence for expressions - because expressions typically have sub-expressions permitted from the same or a more restricted level of the hierarchy, more restricted levels take precedence over less restricted levels. Technically, operator precedence (as described in the Operators section) only applies to determine precedence between operator expressions on the same level of the hierarchy, however the operator precedence table I've provided is effective at describing precedence relationships between different levels of the hierarchy as well.

The 4 levels of the hierarchy (from least to most restrictive) are:

• Statement grammar
• Condition grammar
• Argument grammar
• Primary grammar

##Statement grammar The statement grammar contains expressions that are not permitted to be nested anywhere, because they have unusual syntax structures or interpretations, or have potential to cause syntax ambiguities if nested within other expressions.

Statement grammar expressions are used inside:

• Blocks of statements
• The right-hand-side of a rescue statement modifier (but not an assignment modifier)

Permitted expressions in statement grammar are:

• All condition grammar expressions
• alias/undef/BEGIN{}/END{} expressions
• Assignments where the right-hand-side is a command call (including chained simple assignments i.e. a = b = method arg)
• Assignments with a multiple left- or right-hand-side
• Statement modifiers (if/unless/while/until/rescue)

##Condition grammar The condition grammar is where the command call lives and is used in places where command calls are permitted (generally because it's not the middle of an argument list and so there won't be confusion about commas separating arguments). This is also where the and, or and not operators live, whose most notable characteristic is that they can have command calls as their operands (unlike &&, || and !). Therefore this level of the hierarchy has the full suite of logical operations for conditional expressions.

Condition grammar expressions are used inside:

• The condition expression in if/unless/while/until expressions
• The value to be matched in case expressions
• The value to be iterated in for expressions
• The operands of and, or, not
• The superclass in class expressions
• The object whose singleton class is targeted in class/def expressions
• The operand of defined? if enclosed in parentheses and there is no whitespace preceding it

Permitted expressions in condition grammar are:

• All argument grammar expressions
• Command calls
• and/or/not operators
• ! <command_call> (as a special concession for the ! operator which normally can't take a condition grammar operand)

##Argument grammar

• arg - argument, can't contain unbracketed commas as they would conflict with the outer method/command call. Used for mlhs = arg_value, general operands incl. assignment rhs, arg to "= ... rescue", defined?, call args and mrhs, optional argument value (method/lambda form)
  • can contain primary
  • can contain unary or infix expressions with assignments, operators, ?: (but NOT the ones reserved for expr)
  • can contain and defined? on arg (version in primary has defined? on expr but requires parens)
• primary - restrictive, only used for left side of . :: and [] operators and optional argument value in | ... |
  • can contain atomic values ("literal", "strings", "xstring", "regexp", "words", "qwords", "var_ref", tFID, "backref")
  • array and hash constructors
  • can contain begin bodystmt end
  • can contain "( expr )" (for tLPAREN_ARG)
  • can contain "( compstmt )" (for tLPAREN)
  • can contain scoped constant references (primary_value::CONSTANT or ::CONSTANT)
  • can contain return and yield (yield can be followed by parenth'd args)
  • can contain "not ()" and "not (expr)"
  • can contain "defined? (expr)"
  • can contain operation brace_block, method_call and method_call brace_block
  • can contain lambdas
  • can contain if, unless, while, until, case, for, class, singleton class, module, def, singleton def, break, next, redo, retry command call brace args, do blocks, and chains

Operator conflict resolution Do/brace/bracket/paren conflict resolution

Character processing details - newlines, eof reserved words