Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

CLI improvements + potpourri #143

Closed
wants to merge 60 commits into from

4 participants

Ger Hobbelt Zach Carter Johny Jose Petka Antonov
Ger Hobbelt

Though instead of really pulling this one you might just want to take a quick look at it all and cherrypick, say, commit 79ac482 instead:

  • added a sanity check to the CLI --module-type parameter
  • made sure the CLI --debug parameter is treated as the boolean flag it is

I HOPE, next to that, you go for the git submodules way like I did[*] because node/npm-based 'tracking the latest work in the libraries used by the jison tool' isn't exactly exhilarating when you're working on the [jison] tool itself (and since jison is a development tool, not a production-critical-path component, I think using submodules is a better way than going happy happy joy joy in the node/npm dept.)

[*] I know that bit is graded It Works For Me(tm) so tread carefully. The idea is more important than the actual code here.


Anyway, this should be about the last of the pull req's tonight. Been sitting on the stuff long enough -- at least it got (& gets) tested with a complex production grammar so the most glaring bugs have been removed from the mix.

atrniv and others added some commits
Johny Jose atrniv Fixed an error in topState which used to return the penultimate state…
… instead of the last state on the stack.
08fee30
Elias reduced duplication in lib/jison.js 32bf307
Ger Hobbelt GerHobbelt patched the code generator as it broke on the offending line (where t…
…he action code is poured into a Function()) for action blocks such as this one:

    ... rule ...
    | TAN '(' e ')'
        {
            $$ = {
                token:      FKW_FUNCTION,
                token_attr: FKA_ABSOLUTE,
                childs:     [null, $3],
                loc_info:   @$,
                type:       FT_NUMBER,
                unit:       $3.unit,
                value:      Math.abs($3.value)
            };
        }

The object creation like that would produce a 'Unexpected symbol' error.
47206a3
Ger Hobbelt GerHobbelt and the same 'barf hairball on good action code' problem in the lexer…
…, fixed now. Unfortunately it also means that errorneous generated action code is only detected when you try to run the generated parser.

(I ran into the issue that jison is very whitespace sensitive: the lexer b0rked because a line separating two lexer rules contained a series of spaces and jison then decided that the second rule was simply some more action code of the first :-( )
34b6474
Ger Hobbelt GerHobbelt fix: ensure that ALL tokens are converted to their numeric ID in the …
…lexer actions. Of course the lexer MAY return strings but it is cleaner and faster to return numeric IDs every time.

This resolves the lexer output oddness for larger lexer action blocks where multiple 'return' statements can be placed in conditional sections inside the action block. for example (simplified):

([A-Za-z0-9_][A-Za-z0-9_]*         %{
    console.log("looking up row/column/named identifier token in symbol table: ", yytext, this, this.matches);
    s = yytext.toUpperCase();
    if (yy.symbolTable[s]) {
        rv = yy.symbolTable[s];
        yytext = {
            text: yytext,
            col: function() {
                throw new Exception("I'll be buggered!");
            }
        };
        return rv.token;
    }

    // Would it be a viable basic column identifier?
    match = s.match(/^[A-Z]+$/);
    if (match) {
        // check if this is a legal column id:
        return 'COLUMN_ID';
    }
    // Would it be a viable basic row identifier?
    match = s.match(/^[0-9]+$/);
    if (match) {
        return 'ROW_ID';
    }
    return 'error';
%}
aa1c979
Ger Hobbelt GerHobbelt http://api.jquery.com/live/ : .live() is long gone deprecated. Now it…
… is finally removed and the code had to be changed to use .on() like it should.
62a610a
Ger Hobbelt GerHobbelt add some basic documentation to the generated parser for easier maint…
…ainability / usability of the generated code.
cce66ae
Ger Hobbelt GerHobbelt NONASSOC constant defined twice + some code beautification (which als…
…o makes the generated code a bit more palatable for me) + minimal sync between base code and 'web/assets' copies.
e25dbfb
Ger Hobbelt GerHobbelt Merge remote-tracking branch 'remotes/CatTail/master' 15f40fd
Ger Hobbelt GerHobbelt coding practices 3c0cfaa
Ger Hobbelt GerHobbelt updated jquery copy and added the gh-pages branch as submodule so we …
…can work on both at the same time.
cebf52c
Ger Hobbelt GerHobbelt added the handlebars grammar examples from the gh-pages branch e3dc9e6
Ger Hobbelt GerHobbelt copied the 'relative path' code from gh-pages back to the nanoc sourc…
…es (TBD: get the layout to work for a/b/ subdirs again)

This copying includes the USF page updates (style, content and code) and the TRY page updates (linking to the USF page as that one is pretty useful during development)
b4f5034
Ger Hobbelt GerHobbelt make sure the parseError member function is reset to the default once…
… a previous run of the same instance has temporarily overridden the parseError handler via the parser.yy.parseError override.

sample code (shortened):
  parser.yy.parseError = function local_override() { ... };
  parser.parse(input_a);
  // the next run should use the default error handler again:
  parser.yy.parseError = null;
  parser.parse(input_b);
7b0e636
Ger Hobbelt GerHobbelt outlining (whitespace/braces) 18567d5
Ger Hobbelt GerHobbelt attempt at fixing the prototype bug in the parseError reset logic -- h… e7435a6
Ger Hobbelt GerHobbelt - generated code now includes the lexer methods' descriptions for imp…
…roved readability of the generated code.

- fixes the prototype bug in the parseError reset code: this fix is browser-independent.
4c63ca6
Ger Hobbelt GerHobbelt added support for a 'backtracking' lexer: this is a standard (non-fle…
…x) lexer which for each matching regex will invoke the corresponding action code: when this returns a TOKEN (~ truthy value), then that token is produced (so far the behaviour is identical to the regular lexer); when no token is produced, then the context is rewound and the next matching regex is tried instead, until either a token has been produced or the lexer rules set (regex set) has been exhausted, in which case an ERROR exception is thrown (this last bit is identical to the default lexer too).

This behaviour can be controlled in the generated run-time by the options.backtrack_lexer boolean.
5c92866
Ger Hobbelt GerHobbelt - added support for more() in the backtracking lexer: now when a lexe…
…r rule invokes more() it is assumed to have succeeded, just as when it would have returned a token.

- The parser.lex() method has been corrected to correctly support the new lexer code, which will return boolean FALSE when no token was produced (so that .lexer.js() will recurse until a token has been produced by the lexer action code after all) -- I kept the recursion in there instead of turning it into a loop as it would rather quickly cause a fatal error in the JavaScript engine when the lexer action code is broken (and never produces a token): replacing this recursion by a loop would hang the lexer more or less indefinitely, making debugging such a situation much harder.

- added comments for the lexer condition state methods.

- fixed the topState method similar to commit 32bf307 from @eliasdorneles

- augmented the lexer condition state access methods to improve robustness (e.g. against overzealous use of popState() in user action code)

- topState() now can produce one of the pushed lexer condition states when a (positive, non-zero) numeric index is passed as a parameter: topState(0) is identical to topState() and produces the currently active lexer condition state, while topState(1) produces the previously active lexer condition state, etc.

- minor legibility tweak of the generated lexer code
34130aa
Ger Hobbelt GerHobbelt - lexer: corrected the backtracking feature: the user must call the l…
…exer method .reject() to signal that the current lexer rule FAILED to match a suitable token. (This fixes the scenario where a lexer rule is used to skip whitespace: such a tule won't return a token, nor call more() and should still 'succeed'.)

- lexer: ._less was unused

- lexer: added the ._backtrack boolean member variable to track the invocation of reject() in the lexer action code.
53b6a35
Ger Hobbelt GerHobbelt Merge remote-tracking branch 'remotes/eliasdorneles/master': reduced …
…code duplication

Conflicts:
	lib/jison.js
9535e05
Ger Hobbelt GerHobbelt Merge remote-tracking branch 'remotes/atrniv/master': topState fix, d…
…one another way...

Conflicts:
	lib/jison/lexer.js
a0e648e
Ger Hobbelt GerHobbelt use NodeJS for the tests instead of narwhal f287896
Ger Hobbelt GerHobbelt fixed rule referencing in the new lexer code. 23eb0d1
Ger Hobbelt GerHobbelt remove exception throwing code when rule does not produce a token (th…
…ere are whitespace consuming lexer rules after all which INTENTIONALLY don't produce a token nor do such lexer rules call more() or reject().
569d426
Ger Hobbelt GerHobbelt fixes error recovery logic in the parser run-time (tested & verified …
…using the errorlab.js test file): previously 5 tests would fail, but after this fix, all pass.

The loop which looks for a matching error rule has been abstracted out into the function locateNearestErrorRecoveryRule() because the first cause for failed the tests was the parseError() handler firing before the erorr recovery could kick in: this (and user-defined) parseError handlers need a way to detect whether an error recovery rule is available (via the hash.recoverable boolean).

The tests also uncovered an infinitely loop in error recovery in the new code when the lexer hits EOF. This bug has been fixed.
d48466f
Ger Hobbelt GerHobbelt run 'make'; all tests pass: Passed:79 Failed:0 Errors:0 bf39546
Ger Hobbelt GerHobbelt run 'make site' f483636
Ger Hobbelt GerHobbelt Merge branch 'master' of github.com:GerHobbelt/jison 38d1245
Ger Hobbelt GerHobbelt adjusted the code to remove the error recovery from the run-time when…
… no error recovery rules are included in the user-provided grammar. (The correction was necessary as the run-time has been altered in the previous commits.)
24e36a5
Ger Hobbelt GerHobbelt updated the website pages following the latest changes (verified to w…
…ork using the included tests)
5775f2f
Ger Hobbelt GerHobbelt only preprocess yytext, yyleng and yylineno keywords in action code w…
…hen these are actual entire words.
c181b83
Ger Hobbelt GerHobbelt - lexer.unput(): this.yyleng got out of sync with this.yytext.length …
…as it was not updated after the location range has been updated using its previous value

- ran 'make' and 'make site' to build and test the new code: zero failures.
872b540
Ger Hobbelt GerHobbelt - corrected the backtrack_lexer option name in the run-time exception…
… when reject() is invoked in the action code inside a non-backtracking lexer

- added lexer unit tests to verify proper backtrack_lexer operation
053f899
Ger Hobbelt GerHobbelt - corrected bugs in the backtracking lexer run-time code
- adjusted the backtracking lexer unit tests to ensure the correct behaviour is exhibited for both regular and backtracking lexers (i.e. backtracking errors due to reject() usage in regular lexers must be thrown)

- assert.throw instead of assert["throw"], just like the other assert.xyz NodeJS methods.
2b892e8
Ger Hobbelt GerHobbelt lexer run-time: make sure that too many calls to popState() do not nu…
…ke the lexer condition stack.
538e4fa
Ger Hobbelt GerHobbelt Added fix for when grammar file leaves some (invisible) whitespace on…
… the empty lines between lexer rules: previously, the lexer would, depending on the exact input, barf an inexplicable hairball or (worse!) fail silently. Unit tests to verify the success of the fix have been included.
34233d9
Ger Hobbelt GerHobbelt updated website code ('make site') 4423427
Ger Hobbelt GerHobbelt fixed typo in backtracking lexer code b805599
Ger Hobbelt GerHobbelt Merge branch 'master' of github.com:GerHobbelt/jison 5bb68bd
Ger Hobbelt GerHobbelt regenerated & tested all required files via 'make' + 'make site' e7023d5
Ger Hobbelt GerHobbelt fixed bug in run-time lexer engine when backtracking is enabled but l…
…exer rule did not invoke reject() and did not 'return' a valid token ID: then the rule should be exactly like in the regular lexer: no further rules should be tested and the lexer should simply return false in order for the next bit of input to be lexed. (This happens, for instance, when skipping over whitespace in a backtracking lexer.)
094dc9c
Ger Hobbelt GerHobbelt Merge branch 'master' of github.com:GerHobbelt/jison 76594f0
Ger Hobbelt GerHobbelt Merge branch 'master' of github.com:GerHobbelt/jison 5725a90
Ger Hobbelt GerHobbelt updated the generated files ('make' + 'make test') 8a6971c
Ger Hobbelt GerHobbelt Merge branch 'master' of github.com:GerHobbelt/jison 491304d
Ger Hobbelt GerHobbelt updated the website ('make site') 93b2d96
Ger Hobbelt GerHobbelt missed one spot to update typo yyloc --> yylloc -- probably due to sc…
…rewup in git push/pull as I was working on two platforms at the same time :-( )
b7b7893
Ger Hobbelt GerHobbelt moved the USF images to the pngs/ asset directory + corrected the nan…
…oc rules to ensure the try/usf page is properly regenerated -- that's the page where you can see the diagnostics of the jison run (state machine), so it's quite handy to have around.
57aee14
Ger Hobbelt GerHobbelt the lexer condition state stack was growing indefinitely (2 states pe…
…r rule with action code): the new code keeps the lexer condition state stack sane as now the action state is correctly POPped off the stack when a action code block has been lexed.
7c6eb57
Ger Hobbelt GerHobbelt yylloc is now expanded by the lexer, just like yytext, yylineo and yy…
…leng already were.
640dd9b
Ger Hobbelt GerHobbelt updated Makefile to use the gh-pages self-referential submodule: now …
…the gh-pages are updated immediately when invoking 'make deploy'.
351f865
Ger Hobbelt GerHobbelt updated the Makefile to ensure it does NOT FAIL when the gh-pages git…
… commands find there's nothing to update -- this is important when this Makefile is invoked by other Makefiles: 'nothing to update' is not really an error, after all.
b81a53c
Petka Antonov petkaantonov Add test case to verify bug 07f477f
Petka Antonov petkaantonov Fix off-by-one error f8901b3
Ger Hobbelt GerHobbelt - updated CLI code to work with recent nomnom releases (jison would l…
…ock up when used with a recent nonom version)

- updated the npm install package file: the listed versions are the ones used for testing.

- removed the npm install package dependency on an antiauated copy of jison, which was used to bootstrap this stuff.
764765b
Ger Hobbelt GerHobbelt Merge remote-tracking branch 'remotes/petkaantonov/master'
Conflicts:
	lib/jison/lexer.js
	tests/lexer/regexplexer.js
114732c
Ger Hobbelt GerHobbelt Merge remote-tracking branch 'remotes/zaach-original/master'; also re…
…generated the files and site.

Conflicts:
	Makefile
	lib/jison.js
	lib/jison/cli-wrapper.js
	lib/jison/ebnf.js
	lib/jison/lexer.js
	lib/jison/util/bnf-parser.js
	lib/jison/util/lex-parser.js
	package.json
	src/bnf.l
	src/bnf.y
	src/jisonlex.l
	src/jisonlex.y
	tests/all-tests.js
	tests/grammar/ebnf.js
	tests/grammar/lex_parse.js
	tests/lexer/regexplexer.js
	web/content/assets/js/jison.js
fdb1073
Ger Hobbelt GerHobbelt added the new jison components as submodules f123fdc
Ger Hobbelt GerHobbelt - added a sanity check to the CLI --module-type parameter
- made sure the CLI --debug parameter is treated as the boolean flag it is

- updated the submodules representing the various parts of jison: hopefully the migration of my patches into those buggers went all right...
79ac482
Zach Carter
Owner

npm allows you to link to local versions of packages, which makes developing jison itself pretty seamless. No need to publish or even commit anything before you test changes made to any of the linked dependencies. Much better than submodules, IMO.

Have all of these changes been split into the other PR requests, or are there new things in here as well?

Ger Hobbelt
Ger Hobbelt

Considered closed; another series of edits is due shortly and this has been dealt with via further pull requests, none of which are pending anymore. So we assign this one to the dustbin. ;-)

Ger Hobbelt GerHobbelt closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Oct 1, 2012
  1. Johny Jose

    Fixed an error in topState which used to return the penultimate state…

    atrniv authored
    … instead of the last state on the stack.
Commits on Oct 16, 2012
  1. reduced duplication in lib/jison.js

    Elias authored
Commits on Dec 22, 2012
  1. Ger Hobbelt

    patched the code generator as it broke on the offending line (where t…

    GerHobbelt authored
    …he action code is poured into a Function()) for action blocks such as this one:
    
        ... rule ...
        | TAN '(' e ')'
            {
                $$ = {
                    token:      FKW_FUNCTION,
                    token_attr: FKA_ABSOLUTE,
                    childs:     [null, $3],
                    loc_info:   @$,
                    type:       FT_NUMBER,
                    unit:       $3.unit,
                    value:      Math.abs($3.value)
                };
            }
    
    The object creation like that would produce a 'Unexpected symbol' error.
  2. Ger Hobbelt

    and the same 'barf hairball on good action code' problem in the lexer…

    GerHobbelt authored
    …, fixed now. Unfortunately it also means that errorneous generated action code is only detected when you try to run the generated parser.
    
    (I ran into the issue that jison is very whitespace sensitive: the lexer b0rked because a line separating two lexer rules contained a series of spaces and jison then decided that the second rule was simply some more action code of the first :-( )
Commits on Dec 23, 2012
  1. Ger Hobbelt

    fix: ensure that ALL tokens are converted to their numeric ID in the …

    GerHobbelt authored
    …lexer actions. Of course the lexer MAY return strings but it is cleaner and faster to return numeric IDs every time.
    
    This resolves the lexer output oddness for larger lexer action blocks where multiple 'return' statements can be placed in conditional sections inside the action block. for example (simplified):
    
    ([A-Za-z0-9_][A-Za-z0-9_]*         %{
        console.log("looking up row/column/named identifier token in symbol table: ", yytext, this, this.matches);
        s = yytext.toUpperCase();
        if (yy.symbolTable[s]) {
            rv = yy.symbolTable[s];
            yytext = {
                text: yytext,
                col: function() {
                    throw new Exception("I'll be buggered!");
                }
            };
            return rv.token;
        }
    
        // Would it be a viable basic column identifier?
        match = s.match(/^[A-Z]+$/);
        if (match) {
            // check if this is a legal column id:
            return 'COLUMN_ID';
        }
        // Would it be a viable basic row identifier?
        match = s.match(/^[0-9]+$/);
        if (match) {
            return 'ROW_ID';
        }
        return 'error';
    %}
Commits on Dec 25, 2012
  1. Ger Hobbelt

    http://api.jquery.com/live/ : .live() is long gone deprecated. Now it…

    GerHobbelt authored
    … is finally removed and the code had to be changed to use .on() like it should.
  2. Ger Hobbelt

    add some basic documentation to the generated parser for easier maint…

    GerHobbelt authored
    …ainability / usability of the generated code.
Commits on Dec 26, 2012
  1. Ger Hobbelt

    NONASSOC constant defined twice + some code beautification (which als…

    GerHobbelt authored
    …o makes the generated code a bit more palatable for me) + minimal sync between base code and 'web/assets' copies.
  2. Ger Hobbelt
  3. Ger Hobbelt

    coding practices

    GerHobbelt authored
  4. Ger Hobbelt

    updated jquery copy and added the gh-pages branch as submodule so we …

    GerHobbelt authored
    …can work on both at the same time.
  5. Ger Hobbelt
  6. Ger Hobbelt

    copied the 'relative path' code from gh-pages back to the nanoc sourc…

    GerHobbelt authored
    …es (TBD: get the layout to work for a/b/ subdirs again)
    
    This copying includes the USF page updates (style, content and code) and the TRY page updates (linking to the USF page as that one is pretty useful during development)
  7. Ger Hobbelt

    make sure the parseError member function is reset to the default once…

    GerHobbelt authored
    … a previous run of the same instance has temporarily overridden the parseError handler via the parser.yy.parseError override.
    
    sample code (shortened):
      parser.yy.parseError = function local_override() { ... };
      parser.parse(input_a);
      // the next run should use the default error handler again:
      parser.yy.parseError = null;
      parser.parse(input_b);
  8. Ger Hobbelt
  9. Ger Hobbelt
  10. Ger Hobbelt

    - generated code now includes the lexer methods' descriptions for imp…

    GerHobbelt authored
    …roved readability of the generated code.
    
    - fixes the prototype bug in the parseError reset code: this fix is browser-independent.
Commits on Jan 5, 2013
  1. Ger Hobbelt

    added support for a 'backtracking' lexer: this is a standard (non-fle…

    GerHobbelt authored
    …x) lexer which for each matching regex will invoke the corresponding action code: when this returns a TOKEN (~ truthy value), then that token is produced (so far the behaviour is identical to the regular lexer); when no token is produced, then the context is rewound and the next matching regex is tried instead, until either a token has been produced or the lexer rules set (regex set) has been exhausted, in which case an ERROR exception is thrown (this last bit is identical to the default lexer too).
    
    This behaviour can be controlled in the generated run-time by the options.backtrack_lexer boolean.
  2. Ger Hobbelt

    - added support for more() in the backtracking lexer: now when a lexe…

    GerHobbelt authored
    …r rule invokes more() it is assumed to have succeeded, just as when it would have returned a token.
    
    - The parser.lex() method has been corrected to correctly support the new lexer code, which will return boolean FALSE when no token was produced (so that .lexer.js() will recurse until a token has been produced by the lexer action code after all) -- I kept the recursion in there instead of turning it into a loop as it would rather quickly cause a fatal error in the JavaScript engine when the lexer action code is broken (and never produces a token): replacing this recursion by a loop would hang the lexer more or less indefinitely, making debugging such a situation much harder.
    
    - added comments for the lexer condition state methods.
    
    - fixed the topState method similar to commit 32bf307 from @eliasdorneles
    
    - augmented the lexer condition state access methods to improve robustness (e.g. against overzealous use of popState() in user action code)
    
    - topState() now can produce one of the pushed lexer condition states when a (positive, non-zero) numeric index is passed as a parameter: topState(0) is identical to topState() and produces the currently active lexer condition state, while topState(1) produces the previously active lexer condition state, etc.
    
    - minor legibility tweak of the generated lexer code
  3. Ger Hobbelt

    - lexer: corrected the backtracking feature: the user must call the l…

    GerHobbelt authored
    …exer method .reject() to signal that the current lexer rule FAILED to match a suitable token. (This fixes the scenario where a lexer rule is used to skip whitespace: such a tule won't return a token, nor call more() and should still 'succeed'.)
    
    - lexer: ._less was unused
    
    - lexer: added the ._backtrack boolean member variable to track the invocation of reject() in the lexer action code.
  4. Ger Hobbelt

    Merge remote-tracking branch 'remotes/eliasdorneles/master': reduced …

    GerHobbelt authored
    …code duplication
    
    Conflicts:
    	lib/jison.js
  5. Ger Hobbelt

    Merge remote-tracking branch 'remotes/atrniv/master': topState fix, d…

    GerHobbelt authored
    …one another way...
    
    Conflicts:
    	lib/jison/lexer.js
  6. Ger Hobbelt
  7. Ger Hobbelt
  8. Ger Hobbelt

    remove exception throwing code when rule does not produce a token (th…

    GerHobbelt authored
    …ere are whitespace consuming lexer rules after all which INTENTIONALLY don't produce a token nor do such lexer rules call more() or reject().
Commits on Jan 6, 2013
  1. Ger Hobbelt

    fixes error recovery logic in the parser run-time (tested & verified …

    GerHobbelt authored
    …using the errorlab.js test file): previously 5 tests would fail, but after this fix, all pass.
    
    The loop which looks for a matching error rule has been abstracted out into the function locateNearestErrorRecoveryRule() because the first cause for failed the tests was the parseError() handler firing before the erorr recovery could kick in: this (and user-defined) parseError handlers need a way to detect whether an error recovery rule is available (via the hash.recoverable boolean).
    
    The tests also uncovered an infinitely loop in error recovery in the new code when the lexer hits EOF. This bug has been fixed.
  2. Ger Hobbelt
  3. Ger Hobbelt

    run 'make site'

    GerHobbelt authored
  4. Ger Hobbelt
  5. Ger Hobbelt

    adjusted the code to remove the error recovery from the run-time when…

    GerHobbelt authored
    … no error recovery rules are included in the user-provided grammar. (The correction was necessary as the run-time has been altered in the previous commits.)
  6. Ger Hobbelt

    updated the website pages following the latest changes (verified to w…

    GerHobbelt authored
    …ork using the included tests)
  7. Ger Hobbelt

    only preprocess yytext, yyleng and yylineno keywords in action code w…

    GerHobbelt authored
    …hen these are actual entire words.
  8. Ger Hobbelt

    - lexer.unput(): this.yyleng got out of sync with this.yytext.length …

    GerHobbelt authored
    …as it was not updated after the location range has been updated using its previous value
    
    - ran 'make' and 'make site' to build and test the new code: zero failures.
  9. Ger Hobbelt

    - corrected the backtrack_lexer option name in the run-time exception…

    GerHobbelt authored
    … when reject() is invoked in the action code inside a non-backtracking lexer
    
    - added lexer unit tests to verify proper backtrack_lexer operation
  10. Ger Hobbelt

    - corrected bugs in the backtracking lexer run-time code

    GerHobbelt authored
    - adjusted the backtracking lexer unit tests to ensure the correct behaviour is exhibited for both regular and backtracking lexers (i.e. backtracking errors due to reject() usage in regular lexers must be thrown)
    
    - assert.throw instead of assert["throw"], just like the other assert.xyz NodeJS methods.
Commits on Jan 8, 2013
  1. Ger Hobbelt

    lexer run-time: make sure that too many calls to popState() do not nu…

    GerHobbelt authored
    …ke the lexer condition stack.
Commits on Jan 9, 2013
  1. Ger Hobbelt

    Added fix for when grammar file leaves some (invisible) whitespace on…

    GerHobbelt authored
    … the empty lines between lexer rules: previously, the lexer would, depending on the exact input, barf an inexplicable hairball or (worse!) fail silently. Unit tests to verify the success of the fix have been included.
  2. Ger Hobbelt
Commits on Jan 10, 2013
  1. Ger Hobbelt
  2. Ger Hobbelt
  3. Ger Hobbelt
  4. Ger Hobbelt

    fixed bug in run-time lexer engine when backtracking is enabled but l…

    GerHobbelt authored
    …exer rule did not invoke reject() and did not 'return' a valid token ID: then the rule should be exactly like in the regular lexer: no further rules should be tested and the lexer should simply return false in order for the next bit of input to be lexed. (This happens, for instance, when skipping over whitespace in a backtracking lexer.)
  5. Ger Hobbelt
  6. Ger Hobbelt
  7. Ger Hobbelt
  8. Ger Hobbelt
Commits on Jan 11, 2013
  1. Ger Hobbelt
  2. Ger Hobbelt

    missed one spot to update typo yyloc --> yylloc -- probably due to sc…

    GerHobbelt authored
    …rewup in git push/pull as I was working on two platforms at the same time :-( )
Commits on Jan 15, 2013
  1. Ger Hobbelt

    moved the USF images to the pngs/ asset directory + corrected the nan…

    GerHobbelt authored
    …oc rules to ensure the try/usf page is properly regenerated -- that's the page where you can see the diagnostics of the jison run (state machine), so it's quite handy to have around.
Commits on Jan 21, 2013
  1. Ger Hobbelt

    the lexer condition state stack was growing indefinitely (2 states pe…

    GerHobbelt authored
    …r rule with action code): the new code keeps the lexer condition state stack sane as now the action state is correctly POPped off the stack when a action code block has been lexed.
  2. Ger Hobbelt
  3. Ger Hobbelt

    updated Makefile to use the gh-pages self-referential submodule: now …

    GerHobbelt authored
    …the gh-pages are updated immediately when invoking 'make deploy'.
  4. Ger Hobbelt

    updated the Makefile to ensure it does NOT FAIL when the gh-pages git…

    GerHobbelt authored
    … commands find there's nothing to update -- this is important when this Makefile is invoked by other Makefiles: 'nothing to update' is not really an error, after all.
Commits on Jan 29, 2013
  1. Petka Antonov
  2. Petka Antonov

    Fix off-by-one error

    petkaantonov authored
Commits on Feb 1, 2013
  1. Ger Hobbelt

    - updated CLI code to work with recent nomnom releases (jison would l…

    GerHobbelt authored
    …ock up when used with a recent nonom version)
    
    - updated the npm install package file: the listed versions are the ones used for testing.
    
    - removed the npm install package dependency on an antiauated copy of jison, which was used to bootstrap this stuff.
Commits on Feb 7, 2013
  1. Ger Hobbelt

    Merge remote-tracking branch 'remotes/petkaantonov/master'

    GerHobbelt authored
    Conflicts:
    	lib/jison/lexer.js
    	tests/lexer/regexplexer.js
  2. Ger Hobbelt

    Merge remote-tracking branch 'remotes/zaach-original/master'; also re…

    GerHobbelt authored
    …generated the files and site.
    
    Conflicts:
    	Makefile
    	lib/jison.js
    	lib/jison/cli-wrapper.js
    	lib/jison/ebnf.js
    	lib/jison/lexer.js
    	lib/jison/util/bnf-parser.js
    	lib/jison/util/lex-parser.js
    	package.json
    	src/bnf.l
    	src/bnf.y
    	src/jisonlex.l
    	src/jisonlex.y
    	tests/all-tests.js
    	tests/grammar/ebnf.js
    	tests/grammar/lex_parse.js
    	tests/lexer/regexplexer.js
    	web/content/assets/js/jison.js
  3. Ger Hobbelt
  4. Ger Hobbelt

    - added a sanity check to the CLI --module-type parameter

    GerHobbelt authored
    - made sure the CLI --debug parameter is treated as the boolean flag it is
    
    - updated the submodules representing the various parts of jison: hopefully the migration of my patches into those buggers went all right...
Something went wrong with that request. Please try again.