Lex example doesn't work

[vinipsmaker]

I'm trying to use the rhombus-parser library, but the examples don't work as presented.

I installed the package using:

raco pkg install rhombus-parser-lib
raco pkg install rhombus-parser

The documentation examples make use of a lexer object, but it's not immediately obvious how to import it. I think I got it right by doing the following:

import:
    parser/lex:
        expose lexer

But then the very first example already fails to run:

def lex:
  lexer
  | "he" ["l" "L"]+ "o": [lexeme, position, span]
  | ~eof: #'done

The error:

t.rhm:9:4: ?: misplaced term;
 no infix operator is between this term and the preceding form,
 and `#%literal` is not bound as an implicit prefix operator
  in: "he"
  location...:
   t.rhm:9:4
  context...:
   .../private/parse-interp.rkt:646:50
   /home/vinipsmaker/.local/share/racket/9.1/pkgs/rhombus-lib/rhombus/private/main.rkt:199:2
   .../private/parse-classic.rkt:75:7: parse-:base8
   /home/vinipsmaker/.local/share/racket/9.1/pkgs/rhombus-lib/rhombus/private/expr-macro.rkt:159:7
   /home/vinipsmaker/.local/share/racket/9.1/pkgs/rhombus-lib/rhombus/private/operator.rkt:312:5
   /home/vinipsmaker/.local/share/racket/9.1/pkgs/rhombus-lib/rhombus/private/transform.rkt:20:0: call-as-transformer
   /home/vinipsmaker/.local/share/racket/9.1/pkgs/rhombus-lib/rhombus/private/main.rkt:274:8: dispatch-prefix-operator
   /home/vinipsmaker/.local/share/racket/9.1/pkgs/rhombus-lib/rhombus/private/main.rkt:199:2
   .../private/parse-classic.rkt:75:7: parse-:expression5
   .../private/parse-interp.rkt:506:8
   /usr/share/racket/collects/syntax/parse/private/residual-interp.rkt:458:0
   /usr/share/racket/collects/syntax/parse/private/residual-interp.rkt:113:0
   /usr/share/racket/collects/syntax/parse/private/residual-interp.rkt:219:2: s-and-loop
   /usr/share/racket/collects/syntax/parse/private/residual-interp.rkt:52:4: sk/bind
   /home/vinipsmaker/.local/share/racket/9.1/pkgs/rhombus-lib/rhombus/private/transformer.rkt:124:3
   /home/vinipsmaker/.local/share/racket/9.1/pkgs/rhombus-lib/rhombus/private/transform.rkt:20:0: call-as-transformer
   ...

I've tried to solve the problem on my own, but it just doesn't work.

[usaoc]

The correct way to import such module is to use open:

#lang rhombus/static
import:
  parser/lex open

def lex:
  lexer
  | "he" ["l" "L"]+ "o": [lexeme, position, span]
  | ~eof: #'done

lex(Port.Input.open_string("hello")) // => ["hello", 1, 5]

The reason is that the module provides a sub-language, which needs bindings for implicit forms such as #%literal to work as intended (#%literal is implicitly used when you have literal terms on their own, like the literal string "he"). If you import the module qualified or selectively, without exposing the implicit forms, #%literal and other implicit forms will be unbound when they are needed; this is the error you’re seeing. Now, if the module only provides functions and classes, importing it qualified is definitely a better idea, and the documentation should show that by using a prefix. In the case of parser/lex, the documentation doesn’t use a prefix, which suggests that you should import it open.

Some relevant documentation pages:

• “Namespaces and Modules” in the Rhombus reference explains the prefix convention used in the documentation;
• “Implicit Forms” in the Rhombus metaprogramming tutorial and same titled page in the Rhombus reference discusses implicit forms.

[usaoc]

And, I definitely thought about this a few times: maybe import statements in docs can say open or as <prefix> to suggest the recommended prefix, in addition to the prefix used in the entries?

[mflatt]

I like this idea of shown open in the documentation, so I've added that to Scribble and changed various places in the documentation to use that form.

Adding support for as could makes sense, too — but, in all the cases I've considered so far, it seemed best to pick a module name so that so separate as suggestion would be needed.

[rfindler]

I like this idea a lot, too! I have found this confusing myself (maybe mostly because I just forget that "open" isn't the default ... 😅) so it seems great that this makes the examples more "copy and paste"-able.

[vinipsmaker]

Thanks, @usaoc

[vinipsmaker]

Btw, I wrote an example comparing Python's PLY and Rhombus' lexer:

tokens = (
   'PLUS',
   'NUMBER',
)

t_PLUS    = r'\+'

def t_NUMBER(t):
    r'\d+'
    t.value = int(t.value)    
    return t

def lex:
  lexer
  | "+": "PLUS"
  | digit+: String.to_int(lexeme)