diff --git a/src/ref/lexer.md b/src/ref/lexer.md index 4e84738..ac10d0a 100644 --- a/src/ref/lexer.md +++ b/src/ref/lexer.md @@ -9,10 +9,95 @@ by the lexer, but they separate tokens, e.g., identifiers or literals. ## Comments +Fram supports two kinds of comments: single-line and block comments. Comments +are treated similarly to whitespace: they are ignored by the lexer, but they +always separate tokens. + +Single-line comments start with the `#` character and end with a new line or +the end of file. + +Block comments are introduced by the sequence `{#` followed by any, possibly +empty, sequence *name* of valid identifier characters, operators, and +`#` characters. Such a block comment is terminated by the first occurrence of +*name* that is immediately followed by `#}`. More precisely, it can be +described by the following grammar. +```bnf +block-comment-name ::= { ident-char | op-char | "#" } +block-comment-start ::= "{#" block-comment-name +block-comment-end ::= block-comment-name "#}" +``` +Non-terminal symbols `ident-char` and `op-char` are defined later in this +chapter. At the comment opening, the longest consecutive sequence described by +`block-comment-name` is taken as the comment name. This name should be a suffix +of the name provided at comment closing. Comments using the same name cannot +be nested. This is not an issue in practice, since the programmer can always +choose a unique name to accomplish nesting. + +By convention, single-line comments starting with `##` and block comments +with a name starting with `#` are used as documentation comments, and can be +recognized by some external tools. + +### Examples + +The following code contains some valid comments. +````fram +# This is a single-line comment. +{# This is a block comment. #} +{# Block comments + may span multiple lines. +#} +let id x = x # A single-line comment may appear at the end of a line. + +let n {# A block comment may span a part of a single line. #} = 42 +{#aaa +Comments cannot be nested, +{# but the programmer may choose the comment delimiters. #} +aaa#} + +{#!a! Comment names may contain operators. !a!#} + +{#abc +This comment is ended by `abc` immediately followed by `#}`, +even if the closing sequence is preceded by other characters. +zzabc#} + +let {# +# This is not a single-line comment, +# because comments are not nested. +# This comment can be ended #} here = 13 + +## This is a documentation comment. +let foo x = x + +{## This is another documentation comment. ##} +let bar = foo + +{### +Documentation comments can contain some code +``` +{## with another documentation comment (with a different name). #}} +let some_code = 42 +``` +###} +let baz = bar +```` + ## Literals +```bnf +digit ::= "0"..."9" +lower-char ::= "a"..."z" | "_" +upper-char ::= "A"..."Z" +ident-char ::= lower-char | upper-char | digit | "'" +``` + ## Keywords ## Identifiers ## Operators + +```bnf +op-char ::= "<" | ">" | "&" | "$" | "?" | "!" | "@" | "^" | "+" | "-" + | "~" | "*" | "%" | ";" | "," | "=" | "|" | ":" | "." | "/" +```