lexer.cpp doesn't recognize #define without trailing newline

[dargueta]

Preprocessor statements, at least #define, must have a trailing newline to be recognized.

No trailing newline means it gets expanded into separate tokens:

> s = "#define ASDF"
> for t, v in lexer.cpp(s) do  print(t .. " -> " .. tostring(v))  end
# -> #
iden -> define
iden -> ASDF

With a newline, it's recognized as a preprocessor statement.

> s = "#define ASDF\n"
> for t, v in lexer.cpp(s) do  print(t .. " -> " .. tostring(v))  end
prepro -> #define ASDF

Ideally the behavior would be consistent for both cases. I need to be able to process third-party headers that may or may not end in a newline.

Environment:

• OS: Ubuntu 22.04
• Lua: 5.4.4
• Penlight: 1.13.1

[dargueta]

This is still happening on Penlight 1.14.0:

• OS: Ubuntu 23.10
• Lua: 5.4.6
• Penlight: 1.14.0

[alerque]

Contributions welcome. ;-)