Skip to content

Latest commit

 

History

History
821 lines (566 loc) · 31.3 KB

Rule-Reference.md

File metadata and controls

821 lines (566 loc) · 31.3 KB

Rule Reference

This page contains brief descriptions of all PEGTL rule and combinator classes.

The information about how much input is consumed by the rules only applies to when the rules succeed; the PEGTL is implemented in a way that assumes that rules never consume input when they do not succeed.

Remember that there are two failure modes, only the first of which usually leads to back-tracking:

  • Local failure or a return value of false, in which case the rule must rewind the input to the position at which the rule match was attempted.
  • Global failure or an exception (usually of type tao::parse_error) that is usually generated by a control-class' raise()-method.

Equivalence

Some rule classes are said to be equivalent to a combination of other rules. These rules are not completely equivalent to the shown definition because that is not how they are implemented, therefore:

  • Rule equivalence is with regard to which inputs will match, but:
  • not with regard to which actions will be invoked while matching.

However, rule equivalence does show exactly where the raise<> rule is inserted and therefore which rule will be used to call the control class' raise()-method.

Contents

Meta Rules

These rules are in namespace tao::pegtl.

action< A, R... >
  • Equivalent to seq< R... >, but:
  • Uses the given class template A for actions.
  • Actions can still be disabled explicitly (via disable) or implicitly (via at or not_at).
control< C, R... >
  • Equivalent to seq< R... >, but:
  • Uses the given class template C as control class.
disable< R... >
  • Equivalent to seq< R... >, but:
  • Disables all actions.
discard
  • Equivalent to success, but:
  • Calls the input's discard()-method.
  • See Incremental Input for details.
enable< R... >
  • Equivalent to seq< R... >, but:
  • Enables all actions (if any).
require< Num >
  • Succeeds if at least Num further input bytes are available.
  • With Incremental Input reads the bytes into the buffer.
state< S, R... >
  • Equivalent to seq< R... >, but:
  • Replaces all state arguments with a new instance s of type S.
  • s is constructed with the input and all previous states as arguments.
  • If seq< R... > succeeds then s.success() is called with the input after the match and all previous states as arguments, and, if expected, with A, M, Action, Control as template parameters.

Combinators

Combinators (or combinator rules) are rules that combine (other) rules into new ones.

These are the classical PEG combinator rules defined in namespace tao::pegtl.

at< R... >
  • PEG and-predicate &e
  • Succeeds if and only if seq< R... > would succeed.
  • Consumes nothing, i.e. rewinds after matching.
  • Disables all actions.
  • Allows local failure of R... even within must<> etc.
not_at< R... >
  • PEG not-predicate !e
  • Succeeds if and only if seq< R... > would not succeed.
  • Consumes nothing, i.e. rewinds after matching.
  • Disables all actions.
  • Allows local failure of R... even within must<> etc.
opt< R... >
  • PEG optional e?
  • Optional seq< R... >, i.e. attempt to match seq< R... > and signal success regardless of the result.
  • Equivalent to sor< seq< R... >, success >.
  • Allows local failure of R... even within must<> etc.
plus< R, ... >
  • PEG one-or-more e+
  • Matches seq< R, ... > as often as possible and succeeds if it matches at least once.
  • Equivalent to rep_min< 1, R, ... >.
  • Requires at least one rule R.
seq< R... >
  • PEG sequence e1 e2
  • Sequence or conjunction of rules.
  • Matches the given rules R... in the given order.
  • Fails and stops matching when one of the given rules fails.
  • Consumes everything that the rules R... consumed.
  • Succeeds if no rule is given.
sor< R... >
  • PEG ordered choice e1 / e2
  • Choice or disjunction of rules.
  • Matches the given rules R... in the given order.
  • Succeeds and stops matching when one of the given rules succeeds.
  • Consumes whatever the first rule that succeeded consumed.
  • Allows local failure of R... even within must<> etc.
  • Fails if no rule is given.
star< R, ... >
  • PEG zero-or-more e*
  • Matches seq< R, ... > as often as possible and always succeeds.
  • Allows local failure of R, ... even within must<> etc.
  • Requires at least one rule R.

Convenience

The PEGTL offers a variety of convenience rules which help writing concise grammars as well as offering performance benefits over the equivalent implementation with classical PEG combinators.

These rules are in namespace tao::pegtl.

if_must< R, S... >
  • Attempts to match R and depending on the result proceeds with either must< S... > or failure.
  • Equivalent to seq< R, must< S... > >.
  • Equivalent to if_then_else< R, must< S... >, failure >.
if_must_else< R, S, T >
  • Attempts to match R and depending on the result proceeds with either must< S > or must< T >.
  • Equivalent to if_then_else< R, must< S >, must< T > >.
if_then_else< R, S, T >
  • Equivalent to sor< seq< R, S >, seq< not_at< R >, T > >.
list< R, S >
  • Matches a non-empty list of R separated by S.
  • Equivalent to seq< R, star< S, R > >.
list< R, S, P >
  • Matches a non-empty list of R separated by S where each S can be padded by P.
  • Equivalent to seq< R, star< pad< S, P >, R > >.
list_must< R, S >
  • Matches a non-empty list of R separated by S.
  • Similar to list< R, S >, but if there is an S it must be followed by an R.
  • Equivalent to seq< R, star< if_must< S, R > > >.
list_must< R, S, P >
  • Matches a non-empty list of R separated by S where each S can be padded by P.
  • Similar to list< R, S, P >, but if there is an S it must be followed by an R.
  • Equivalent to seq< R, star< if_must< pad< S, P >, R > > >.
list_tail< R, S >
  • Matches a non-empty list of R separated by S with optional trailing S.
  • Equivalent to seq< list< R, S >, opt< S > >.
list_tail< R, S, P >
  • Matches a non-empty list of R separated by S with optional trailing S and padding P inside the list.
  • Equivalent to seq< list< R, S, P >, opt< star< P >, S > >.
minus< M, S >
  • Succeeds if M matches, and then S does not match all of the input that M matched.
  • Does not call actions for S (unless S contains enable<>).
  • Ignores S for the grammar analysis.
must< R... >
  • Equivalent to seq< R... >, but:
  • Converts local failure of R... into global failure.
  • Calls raise< R > for the R that failed.
  • Equivalent to seq< sor< R, raise< R > >... >.
pad< R, S, T = S >
  • Matches an R that can be padded by arbitrary many S on the left and T on the right.
  • Equivalent to seq< star< S >, R, star< T > >.
pad_opt< R, P >
  • Matches an optional R that can be padded by arbitrary many P or just arbitrary many P.
  • Equivalent to seq< star< P >, opt< R, star< P > > >.
rep< Num, R... >
  • Matches seq< R... > for Num times without checking for further matches.
  • Equivalent to seq< seq< R... >, ..., seq< R... > > where seq< R... > is repeated Num times.
rep_max< Max, R... >
  • Matches seq< R... > for at most Max times and verifies that it doesn't match more often.
  • Equivalent to rep_min_max< 0, Max, R... >.
rep_min< Min, R, ... >
  • Matches seq< R, ... > as often as possible and succeeds if it matches at least Min times.
  • Equivalent to seq< rep< Min, R, ... >, star< R, ... > >.
  • Requires at least one rule R.
rep_min_max< Min, Max, R... >
  • Matches seq< R... > for Min to Max times and verifies that it doesn't match more often.
  • Equivalent to seq< rep< Min, R... >, rep_opt< Max - Min, R... >, not_at< R... > >.
rep_opt< Num, R... >
  • Matches seq< R... > for zero to Num times without check for further matches.
  • Equivalent to rep< Num, opt< R... > >.
star_must< R, S... >
  • Equivalent to star< if_must< R, S... > >.
try_catch< R... >
  • Equivalent to seq< R... >, but:
  • Converts global failure (exception) into local failure (return value false).
  • Catches exceptions of type tao::pegtl::parse_error.
try_catch_type< E, R... >
  • Equivalent to seq< R... >, but:
  • Converts global failure (exception) into local failure (return value false).
  • Catches exceptions of type E.
until< R >
  • Consumes all input until R matches.
  • Equivalent to until< R, any >.
until< R, S, ... >
  • Matches seq< S, ... > as long as at< R > does not match and succeeds when R matches.
  • Equivalent to seq< star< not_at< R >, not_at< eof >, S, ... >, R >.

Action Rules

These rules are in namespace tao::pegtl.

These rules replicate the intrusive way actions were called from within the grammar in the PEGTL 0.x with the apply<> and if_apply<> rules. The actions for these rules are classes (rather than class templates as required for the parse()-functions and action<>-rule). These rules respect the current apply_mode, but neither use the control-class to invoke the actions, nor support actions that return bool.

apply< A... >
  • Equivalent to success wrt. parsing, but also:
  • Calls A::apply() for all A, in order, with an empty input and all states as arguments.
apply0< A... >
  • Equivalent to success wrt. parsing, but also:
  • Calls A::apply0() for all A, in order, with all states as arguments.
if_apply< R, A... >
  • Equivalent to R wrt. parsing, but also:
  • If R matches, calls A::apply(), for all A, in order, with the input matched by R and all states as arguments.

Atomic Rules

These rules are in namespace tao::pegtl.

Atomic rules do not rely on other rules.

bof
  • Succeeds at "beginning-of-file", i.e. when the input's byte() method returns zero.
  • Does not consume input.
  • Does not work with inputs that don't have a byte() method.
bol
  • Succeeds at "beginning-of-line", i.e. when the input's byte_in_line() method returns zero.
  • Does not consume input.
  • Does not work with inputs that don't have a byte_in_line() method.
bytes< Num >
  • Succeeds when the input contains at least Num further bytes.
  • Consumes these Num bytes from the input.
eof
  • Succeeds at "end-of-file", i.e. when the input is empty or all input has been consumed.
  • Does not consume input.
failure
  • Dummy rule that never succeeds.
  • Does not consume input.
raise< T >
  • Generates a global failure.
  • Calls the control-class' Control< T >::raise()-method.
  • T can be a rule, but it does not have to be a rule.
  • Does not consume input.
success
  • Dummy rule that always succeeds.
  • Does not consume input.

ASCII Rules

These rules are in the inline namespace tao::pegtl::ascii.

The ASCII rules operate on single bytes, without restricting the range of values to 7 bits. They are compatible with input with the 8th bit set in the sense that nothing breaks in their presence. Rules like ascii::any or ascii::not_one< 'a' > will match all possible byte values, and all possible byte values excluding 'a', respectively. However the character class rules like ascii::alpha only match the corresponding ASCII characters.

(It is possible to match UTF-8 multi-byte characters with the ASCII rules, for example the Euro sign code point U+20AC, which is encoded by the UTF-8 sequence E2 82 AC, can be matched by either tao::pegtl::ascii::string< 0xe2, 0x82, 0xac > or tao::pegtl::utf8::one< 0x20ac >.)

alnum
  • Matches and consumes a single ASCII alphabetic or numeric character.
  • Equivalent to ranges< 'a', 'z', 'A', 'Z', '0', '9' >.
alpha
  • Matches and consumes a single ASCII alphabetic character.
  • Equivalent to ranges< 'a', 'z', 'A', 'Z' >.
any
  • Matches and consumes any single byte, including all ASCII characters.
  • Equivalent to bytes< 1 >.
blank
  • Matches and consumes a single ASCII horizontal space or horizontal tabulator character.
  • Equivalent to one< ' ', '\t' >.
digit
  • Matches and consumes a single ASCII decimal digit character.
  • Equivalent to range< '0', '9' >.
eol
  • Depends on the Eol template parameter of the input, by default:
  • Matches and consumes a Unix or MS-DOS line ending, that is:
  • Equivalent to sor< one< '\n' >, string< '\r', '\n' > >.
eolf
  • Equivalent to sor< eof, eol >.
identifier_first
  • Matches and consumes a single ASCII character permissible as first character of a C identifier.
  • Equivalent to ranges< 'a', 'z', 'A', 'Z', '_' >.
identifier_other
  • Matches and consumes a single ASCII character permissible as subsequent character of a C identifier.
  • Equivalent to ranges< 'a', 'z', 'A', 'Z', '0', '9', '_' >.
identifier
  • Matches and consumes an ASCII identifier as defined for the C programming language.
  • Equivalent to seq< identifier_first, star< identifier_other > >.
istring< C, ... >
  • Matches and consumes the given ASCII string C, ... with case insensitive matching.
  • Similar to string< C, ... >, but:
  • For ASCII letters a-z and A-Z the match is case insensitive.
keyword< C, ... >
  • Matches and consumes a non-empty string not followed by a subsequent identifier character.
  • Equivalent to seq< string< C, ... >, not_at< identifier_other > >.
lower
  • Matches and consumes a single ASCII lower-case alphabetic character.
  • Equivalent to range< 'a', 'z' >.
not_one< C, ... >
  • Succeeds when the input is not empty, and:
  • The next input byte is not one of C, ....
  • Consumes one byte when it succeeds.
not_range< C, D >
  • Succeeds when the input is not empty, and:
  • The next input byte is not in the closed range C ... D.
  • Consumes one byte when it succeeds.
nul
  • Matches and consumes an ASCII nul character.
  • Equivalent to one< 0 >.
one< C, ... >
  • Succeeds when the input is not empty, and:
  • The next input byte is one of C, ....
  • Consumes one byte when it succeeds.
print
  • Matches and consumes any single ASCII character traditionally defined as printable.
  • Equivalent to range< 32, 126 >.
range< C, D >
  • Succeeds when the input is not empty, and:
  • The next input byte is in the closed range C ... D.
  • Consumes one byte when it succeeds.
ranges< C1, D1, C2, D2, ... >
  • Equivalent to sor< range< C1, D1 >, range< C2, D2 >, ... >.
ranges< C1, D1, C2, D2, ..., E >
  • Equivalent to sor< range< C1, D1 >, range< C2, D2 >, ..., one< E > >.
seven
  • Matches and consumes any single true ASCII character that fits into 7 bits.
  • Equivalent to range< 0, 127 >.
shebang
  • Equivalent to seq< string< '#', '!' >, until< eolf > >.
space
  • Matches and consumes a single space, line-feed, carriage-return, horizontal-tab, vertical-tab or form-feed.
  • Equivalent to one< ' ', '\n', '\r', 't', '\v', '\f' >.
string< C1, C2, ... >
  • Matches and consumes a string, a sequence of bytes or single-byte characters.
  • Equivalent to seq< one< C1 >, one< C2 >, ... >.
TAOCPP_PEGTL_ISTRING( "..." )
  • Macro where TAOCPP_PEGTL_ISTRING( "foo" ) yields
    istring< 'f', 'o', 'o' >.
  • The argument must be a string literal.
  • Works for strings up to 512 bytes of length (excluding trailing '\0').
  • Strings may contain embedded '\0'.
TAOCPP_PEGTL_KEYWORD( "..." )
  • Macro where TAOCPP_PEGTL_KEYWORD( "foo" ) yields
    keyword< 'f', 'o', 'o' >.
  • The argument must be a string literal.
  • Works for keywords up to 512 bytes of length (excluding trailing '\0').
  • Strings may contain embedded '\0'.
TAOCPP_PEGTL_STRING( "..." )
  • Macro where TAOCPP_PEGTL_STRING( "foo" ) yields
    string< 'f', 'o', 'o' >.
  • The argument must be a string literal.
  • Works for strings up to 512 bytes of length (excluding trailing '\0').
  • Strings may contain embedded '\0'.
two< C >
  • Succeeds when the input contains at least two bytes, and:
  • These two input bytes are both C.
  • Consumes two bytes when it succeeds.
upper
  • Matches and consumes a single ASCII upper-case alphabetic character.
  • Equivalent to range< 'A', 'Z' >.
xdigit
  • Matches and consumes a single ASCII hexadecimal digit character.
  • Equivalent to ranges< '0', '9', 'a', 'f', 'A', 'F' >.

UTF-8 Rules

These rules are in namespace tao::pegtl::utf8.

A unicode code point is considered valid when it is in the range 0 to 0x10ffff.

any
  • Succeeds when the input is not empty, and:
  • The next 1-4 bytes are the UTF-8 encoding of a valid unicode code point.
  • Consumes the 1-4 bytes when it succeeds.
bom
  • Succeeds when the input is not empty, and:
  • The next 3 bytes are the UTF-8 encoding of character U+FEFF, byte order mark (BOM).
  • Equivalent to one< 0xfeff >.
not_one< C, ... >
  • Succeeds when the input is not empty, and:
  • The next 1-4 bytes are the UTF-8 encoding of a valid unicode code point, and:
  • The input code point is not one of the given code points C, ....
  • Consumes the 1-4 bytes when it succeeds.
not_range< C, D >
  • Succeeds when the input is not empty, and:
  • The next 1-4 bytes are the UTF-8 encoding of a valid unicode code point, and:
  • The input code point B satisfies B < C || D < B.
  • Consumes the 1-4 bytes when it succeeds.
one< C, ... >
  • Succeeds when the input is not empty, and:
  • The next 1-4 bytes are the UTF-8 encoding of a valid unicode code point, and:
  • The input code point is one of the given code points C, ....
  • Consumes the 1-4 bytes when it succeeds.
range< C, D >
  • Succeeds when the input is not empty, and:
  • The next 1-4 bytes are the UTF-8 encoding of a valid unicode code point, and:
  • The input code point B satisfies C <= B && B <= D.
  • Consumes the 1-4 bytes when it succeeds.
ranges< C1, D1, C2, D2, ... >
  • Equivalent to sor< range< C1, D1 >, range< C2, D2 >, ... >.
ranges< C1, D1, C2, D2, ..., E >
  • Equivalent to sor< range< C1, D1 >, range< C2, D2 >, ..., one< E > >.
string< C1, C2, ... >
  • Equivalent to seq< one< C1 >, one< C2 >, ... >.

UTF-16 Rules

These rules are in namespace tao::pegtl::utf16.

The UTF-16 rules are surrogate-pair-aware and will consume 4 bytes for a single matched code point, rather than 2, whenever a valid surrogate pair is detected. Following what appears to be "best" practice, it is not an error when a code unit in the range 0xd800 to 0xdfff is encountered outside of a valid surrogate pair.

UTF-16 support should be considered experimental and the following limitations apply to the UTF-16 rules:

  • Native byte order is assumed for the input.
  • Unaligned input leads to unaligned memory access.
  • The line and column numbers are not counted correctly.

Unaligned memory is no problem on x86 compatible processors; on some other architectures like ARM an unaligned access will crash the application.

any
  • Succeeds when the input contains at least 2 bytes, and:
  • The next 2 (or 4) input bytes encode a valid unicode code point.
  • Consumes these 2 (or 4) bytes when it succeeds.
bom
  • Succeeds when the input is not empty, and:
  • The next 2 bytes are the UTF-16 encoding of character U+FEFF, byte order mark (BOM).
  • Equivalent to one< 0xfeff >.
not_one< C, ... >
  • Succeeds when the input contains at least 2 bytes, and:
  • The next 2 (or 4) input bytes encode a valid unicode code point, and:
  • The input code point is not one of the given code points C, ....
  • Consumes these 2 (or 4) bytes when it succeeds.
not_range< C, D >
  • Succeeds when the input contains at least 2 bytes, and:
  • The next 2 (or 4) input bytes encode a valid unicode code point, and:
  • The input code point B satisfies B < C || D < B.
  • Consumes these 2 (or 4) bytes when it succeeds.
one< C, ... >
  • Succeeds when the input contains at least 2 bytes, and:
  • The next 2 (or 4) input bytes encode a valid unicode code point, and:
  • The input code point is one of the given code points C, ....
  • Consumes these 2 (or 4) bytes when it succeeds.
range< C, D >
  • Succeeds when the input contains at least 2 bytes, and:
  • The next 2 (or 4) input bytes encode a valid unicode code point, and:
  • The input code point B satisfies C <= B && B <= D.
  • Consumes these 2 (or 4) bytes when it succeeds.
ranges< C1, D1, C2, D2, ... >
  • Equivalent to sor< range< C1, D1 >, range< C2, D2 >, ... >.
ranges< C1, D1, C2, D2, ..., E >
  • Equivalent to sor< range< C1, D1 >, range< C2, D2 >, ..., one< E > >.
string< C1, C2, ... >
  • Equivalent to seq< one< C1 >, one< C2 >, ... >.

UTF-32 Rules

These rules are in namespace tao::pegtl::utf32.

UTF-32 support should be considered experimental and the following limitations apply to the UTF-32 rules:

  • Native byte order is assumed for the input.
  • Unaligned input leads to unaligned memory access.
  • The line and column numbers are not counted correctly.

Unaligned memory is no problem on x86 compatible processors; on some other architectures like ARM an unaligned access will crash the application.

any
  • Succeeds when the input contains at least 4 bytes, and:
  • The next 4 input bytes encode a valid unicode code point.
  • Consumes these 4 bytes when it succeeds.
bom
  • Succeeds when the input is not empty, and:
  • The next 4 bytes are the UTF-32 encoding of character U+FEFF, byte order mark (BOM).
  • Equivalent to one< 0xfeff >.
not_one< C, ... >
  • Succeeds when the input contains at least 4 bytes, and:
  • The next 4 input bytes encode a valid unicode code point, and:
  • The input code point is not one of the given code points C, ....
  • Consumes these 4 bytes when it succeeds.
not_range< C, D >
  • Succeeds when the input contains at least 4 bytes, and:
  • The next 4 input bytes encode a valid unicode code point, and:
  • The input code point B satisfies B < C || D < B.
  • Consumes these 4 bytes when it succeeds.
one< C, ... >
  • Succeeds when the input contains at least 4 bytes, and:
  • The next 4 input bytes encode a valid unicode code point, and:
  • The input code point is one of the given code points C, ....
  • Consumes these 4 bytes when it succeeds.
range< C, D >
  • Succeeds when the input contains at least 4 bytes, and:
  • The next 4 input bytes encode a valid unicode code point, and:
  • The input code point B satisfies C <= B && B <= D.
  • Consumes these 4 bytes when it succeeds.
ranges< C1, D1, C2, D2, ... >
  • Equivalent to sor< range< C1, D1 >, range< C2, D2 >, ... >.
ranges< C1, D1, C2, D2, ..., E >
  • Equivalent to sor< range< C1, D1 >, range< C2, D2 >, ..., one< E > >.
string< C1, C2, ... >
  • Equivalent to seq< one< C1 >, one< C2 >, ... >.

Full Index

Copyright (c) 2014-2017 Dr. Colin Hirsch and Daniel Frey