Stuck at step 0

[matthew-dean]

Alright, I've done the following:

1. Installed nearley via npm.
2. Created a parser.ne file.
3. Copied and pasted an example from the examples folder here.
4. Compiled that example to a grammar JS file successfully. (Although the javascript.ne file does not compile for me.)
5. Run valid input through the parser using the code example in the documentation.

Result:
Error at character 0

Since I'm using all the code provided here, I'm perplexed as to what the problem could be. Even a simple, one-line grammar file fails to actually parse anything. Sooo..... o_O ?

[matthew-dean]

Okay, I finally got it to work with this:

parser.ne

letter -> [a-zA-Z]:+

grammar.js

// Generated automatically by nearley
// http://github.com/Hardmath123/nearley
(function () {
function id(x) {return x[0]; }
var grammar = {
    ParserRules: [
    {"name": "letter", "symbols": [" ebnf$1"]},
    {"name": " ebnf$1", "symbols": [/[a-zA-Z]/]},
    {"name": " ebnf$1", "symbols": [/[a-zA-Z]/, " ebnf$1"], "postprocess": function (d) {
                    return [d[0]].concat(d[1]);
                }}
]
  , ParserStart: "letter"
}
if (typeof module !== 'undefined'&& typeof module.exports !== 'undefined') {
   module.exports = grammar;
} else {
   window.grammar = grammar;
}
})();

Input: aaa
Output: a,a,a

So maybe the problem is simply that none of the .ne files actually work with the current version of nearley? Or maybe I keep pasting in code they don't actually recognize? I tried single-line Lua statements with no success.

[kach]

Hmm, I'd need more code samples to try and guess what the issue is.

[matthew-dean]

To be fair, I'm not a Lua programmer, so there could be code constructs needed there that I'm not aware of. I was cutting / pasting different samples. For the CSV, I tried something like "foo","bar" which should be valid, but maybe isn't by that grammar. The JavaScript one I would've been more confident about input, but I couldn't get it to compile.

The javascript.ne error is:

/usr/local/lib/node_modules/nearley/lib/compile.js:114
    while (i<structure.length) {
                      ^
TypeError: Cannot read property 'length' of undefined
    at Compile (/usr/local/lib/node_modules/nearley/lib/compile.js:114:23)
    at StreamWrapper.<anonymous> (/usr/local/lib/node_modules/nearley/bin/nearleyc.js:44:11)
    at StreamWrapper.emit (events.js:117:20)
    at finishMaybe (_stream_writable.js:360:12)
    at endWritable (_stream_writable.js:367:3)
    at StreamWrapper.Writable.end (_stream_writable.js:345:5)
    at ReadStream.onend (_stream_readable.js:502:10)
    at ReadStream.g (events.js:180:16)
    at ReadStream.emit (events.js:117:20)
    at _stream_readable.js:944:16

[kach]

Weird… I don't think the format has changed enough to outdate those examples. Have you tried using the precompiled versions in examples/js/?

[matthew-dean]

Did just now. The precompiled javascript.js works just fine. (Well, parses anyway.)

[kach]

Weeeeeeird. Freshly-compiled versions of the Lua grammar and the JS grammar work perfectly for me.

$ nearleyc examples/lua.ne > ~/Desktop/test/lua.js
$ echo "a = 42" | nearley-test ~/Desktop/test/lua.js
[junk ommitted]
[ [ [],
    [ [ [ [ [ { name: 'a' } ] ],
          undefined,
          '=',
          undefined,
          [ [ [ [ [ [ [ [ { literal: 42 } ] ] ] ] ] ] ] ] ] ] ],
    undefined ] ]

[matthew-dean]

Like I said, maybe all my cut-and-pasted lua examples sucked somehow. But definitely javascript.ne was throwing an error. What about node version? Might that be an issue? Also, I did get some warnings when installing dependencies for nearley.

npm WARN unmet dependency /usr/local/lib/node_modules/npm/node_modules/normalize-package-data requires github-url-from-git@'~1.1.1' but will load
npm WARN unmet dependency /usr/local/lib/node_modules/npm/node_modules/github-url-from-git,
npm WARN unmet dependency which is version 1.4.0
npm WARN unmet dependency /usr/local/lib/node_modules/npm/node_modules/normalize-package-data requires github-url-from-username-repo@'^0.2.0' but will load
npm WARN unmet dependency /usr/local/lib/node_modules/npm/node_modules/github-url-from-username-repo,
npm WARN unmet dependency which is version 1.0.0

[kach]

I believe those warnings are generic npm warnings being thrown about non-nearley modules (npm thinks it's a good idea to warn you about them even though you don't care about them right now). I tried reinstalling nearley, and I got a completely different set of such warnings. Nothing to worry about.

I'm using nodejs for all testing. If your testing is happening in the browser, then things are slightly different because namespacing. What's your test setup like?

[matthew-dean]

Sorry for the lag in reply. My test setup is using node-webkit, so yes, the nearley module is running in a node context, and I'm just using the webkit part so I can put in new input and keep testing.

I think the hardest part is that if there's any error in parsing anywhere, it simply says "error at character 0". There's no information about where, in the set of grammar, it might cause a parsing error. I mean, I guess I'm assuming it's a parsing error of the input, and not simply that the input doesn't match. Either way, the error doesn't help in figuring out what to do next.

[matthew-dean]

I'm trying to write a grammar parser for Less (Lesscss.org), and I'm starting with the documented grammar for CSS, and then adapting it to nearley. I'm just going to post what I got.

stylesheet  -> ( CDO | CDC | S | statement ):*
statement   -> ruleset | atrule
atrule     -> ATKEYWORD S:* any:* ( block | ";" S:* )
block       -> "{" S:* ( any | block | ATKEYWORD S:* | ";" S:* ):* "}" S:*
ruleset     -> selector? "{" S:* declaration? ( ";" S:* declaration? ):* "}" S:*
selector    -> any:+
declaration -> property ":" S:* value
property    -> IDENT S:*
value       -> ( any | block | ATKEYWORD S:* ):+
any         -> ( IDENT | NUMBER | PERCENTAGE | DIMENSION | STRING
              | DELIM | URI | HASH | UNICODE_RANGE | INCLUDES
              | FUNCTION S:* any:* ")" | DASHMATCH | "(" S:* any:* ")"
              | "[" S:* any:* "]" ) S:*

IDENT -> ident
ATKEYWORD -> "@" ident
STRING  -> string
HASH  -> "#" name
NUMBER  -> num
PERCENTAGE  -> num "%"
DIMENSION -> num ident
URI -> "url(" w (string | urlchar:* ) w ")"
UNICODE_RANGE -> "U+" hexupper ("-" hexupper):?
CDO -> "<!--"
CDC -> "-->"
S -> wc:+
COMMENT -> "/*" [^*]:* "*":+ ([^/] [^*]:* "*":+):* "/"
FUNCTION  -> ident "("
INCLUDES  -> "~="
DASHMATCH -> "|="
PREFIXMATCH -> "^="
SUFFIXMATCH -> "$="
SUBSTRINGMATCH  -> "*="
BOM -> [\uFEFF]


ident -> "-":? nmstart nmchar:*
name  -> nmchar:+
nmstart -> [a-zA-Z] | "_" | nonascii | escape
nonascii  -> [\u0080-\uD7FF\uE000-\uFFFD] # ignoring x10000-x10FFFF from spec, since JS doesn't support it natively
unicode -> "\\" hexlower wc?
hexc -> [0-9a-fA-F]
hexU -> [0-9A-F]
hexlower -> hexc | hexc hexc hexc hexc hexc hexc
hexupper -> hexU | hexU hexU hexU hexU hexU hexU 
escape  -> unicode | "\\" [\u0020-\u007E\u0080-\uD7FF\uE000-\uFFFD] #x10000-x10FFFF in spec
nmchar  -> [a-zA-Z0-9] | "-" | "_" | nonascii | escape
num -> [0-9]:+ | [0-9]:* "." [0-9]:+
string  -> "\"" (stringchar | "'"):* "\"" | "'" (stringchar | "\""):* "'"
stringchar  -> urlchar | [\u0020] | "\"" nl
urlchar -> [\u0009\u0021\u0023-\u0026\u0027-\u007E] | nonascii | escape
nl  -> [\u0A] | [\u0D] [\u0A] | [\u0D] | [\u0C]
w -> wc:*
wc  -> [\u09] | [\u0A] | [\u0C] | [\u0D] | [\u20]

It compiles successfully to grammar.js, but thus far I haven't been able to get it to recognize any valid CSS. Are you able to help me identify the problem? Much appreciated.

[kach]

I think the hardest part is that if there's any error in parsing anywhere, it simply says "error at character 0". There's no information about where, in the set of grammar, it might cause a parsing error. I mean, I guess I'm assuming it's a parsing error of the input, and not simply that the input doesn't match. Either way, the error doesn't help in figuring out what to do next.

Actually, it does mean that the input doesn't match.

Your problem might be that you're using 2-digit unicode values. Try adding 00 before all of them (so, for example, [\u0009] | [\u000A] | [\u000C] | [\u000D] | [\u0020] in the last line).

Similarly, I'm not sure what [\u0020-\u007E\u0080-\uD7FF\uE000-\uFFFD] is trying to accomplish—if you're trying to do character ranges, I'm not sure it works that way. You should probably try these regexes on simple examples in your node console first.