TypeError on collectCandidates

[HelgeG]

Hi,

I am trying to start a project to try out the c3 engine, but I am running into a little obstacle early on. I am setting up the scaffold as outlined in the Getting Started section of the README, but I get the following error when I do the core.collectCandidates(0) call:

TypeError: Cannot read property 'index' of undefined
    at CodeCompletionCore.collectCandidates (/home/helgeg/proj/spotqa/virtuoso/standalone_parser/node_modules/antlr4-c3/src/CodeCompletionCore.ts:107:40)

I am assuming that there is some initialization that I haven't done, but I cannot figure it out. Below is the setup leading up to the failing call.

const antlr4ts = require('antlr4ts/index');
const c3 = require('antlr4-c3');
const ExprParser = require("./ExprParser").ExprParser
const ExprLexer = require("./ExprLexer").ExprLexer
...
let inputStream = new antlr4ts.ANTLRInputStream("var c = a + b()");
let lexer = new ExprLexer(inputStream);
let tokenStream = new antlr4ts.CommonTokenStream(lexer);
let parser = new ExprParser(tokenStream);
let errorListener = new VirtuosoErrorListener();
parser.addErrorListener(errorListener);
let tree = parser.expression();
let core = new c3.CodeCompletionCore(parser);
let candidates = core.collectCandidates(0);

This is using a parser generated by antlr 4.7.1 with a JavaScript target and antlr4-c3 1.1.8. Hopefully someone can glance at the code and tell me where I am going astray.

Stepping through the code, it looks like when I call c3.CodeCompletionCore(parser) the member variables get set (including this.parser = parser;), but when I call core.collectCandidates, the parser.inputStream variable is undefined:

let tokenStream = this.parser.inputStream;  // this.parser.inputStream is undefined
let currentIndex = tokenStream.index; // this fails, since there is no tokenStream

It looks to me like I am not setting the tokenStream correctly since there is no definition of the inputStream on the parser object.

[HelgeG]

I created a "pure" TypeScript project and used antlr4ts-cli to create the parser and lexer classes, and I am not seeing this issue.

[kaidjohnson]

I am also seeing this same issue when using the antlr4 js package with a JavaScript antlr target.

Using:

• antlr4 4.7.2
• anltr4-cs 1.1.11

My grammar test (based on the docs):

grammar Expr;
expression: assignment | simpleExpression;

assignment: (VAR | LET) ID EQUAL simpleExpression;

simpleExpression
    : simpleExpression (PLUS | MINUS) simpleExpression
    | simpleExpression (MULTIPLY | DIVIDE) simpleExpression
    | variableRef
    | functionRef
;

variableRef: ID;
functionRef: ID OPEN_PAR CLOSE_PAR;

VAR: [vV] [aA] [rR];
LET: [lL] [eE] [tT];

PLUS: '+';
MINUS: '-';
MULTIPLY: '*';
DIVIDE: '/';
EQUAL: '=';
OPEN_PAR: '(';
CLOSE_PAR: ')';
ID: [a-zA-Z] [a-zA-Z0-9_]*;
WS: [ \n\r\t] -> channel(HIDDEN);

The grammar is compiled using antlr4 -Dlanguage=JavaScript Expr.g4 per https://github.com/antlr/antlr4/blob/master/doc/javascript-target.md

My code (based on the docs):

import antlr4 from 'antlr4';
import * as c3 from 'antlr4-c3';

const suggestExpression = (input) => {
	const inputStream = new antlr4.InputStream(input);
	const lexer = new ExprLexer(inputStream);
	const tokenStream = new antlr4.CommonTokenStream(lexer);
	const parser = new ExprParser(tokenStream);
	const errorListener = new antlr4.error.ErrorListener();
	parser.addErrorListener(errorListener);

	const tree = parser.expression();
	const core = new c3.CodeCompletionCore(parser);
	return core.collectCandidates(0);
};

The error:

CodeCompletionCore.js:67 Uncaught TypeError: Cannot read property 'index' of undefined
    at CodeCompletionCore.collectCandidates (CodeCompletionCore.js:67)

The lines:

I've tried adding the inputStream to the parser via parser.inputStream, but this results in an infinite loop in the while (true) block just below. It seems that the tokenStream.LT(offset++) call is returning -1 without a .type so the loop just keeps going.

Is antlr4ts required? Seems like using the antlr4 lib should work, too.

[kaidjohnson]

I modified my parser manually and was able to prevent the errors from being thrown. It appears that the inputStream might be the tokenStream, so I attached that to the parser. The ATN states don't have a getTransitions() method on them, even though c3 expects to always retrieve transitions this way, so I mapped a getTransitions method onto each state which returns the state.transitions that are defined.

const suggestExpression = (input) => {
	const inputStream = new antlr4.InputStream(input);
	const lexer = new ExprLexer(inputStream);
	const tokenStream = new antlr4.CommonTokenStream(lexer);

	const parser = new ExprParser(tokenStream);
	const errorListener = new antlr4.error.ErrorListener();
	parser.addErrorListener(errorListener);
	parser.inputStream = tokenStream;
	parser.atn.states.forEach((state) => {
		state.getTransitions = () => state.transitions;
	});

	const tree = parser.expression();
	const core = new c3.CodeCompletionCore(parser);
	return core.collectCandidates(input - 1);
};

I think this is a bit of progress, however I'm still getting 0 candidates back. I assume this has to do with the lack of CodeCompletionCore.preferredRules defined, but the test suggests otherwise:

            let inputStream = new ANTLRInputStream("var c = a + b()");
            let lexer = new ExprLexer(inputStream);
            let tokenStream = new CommonTokenStream(lexer);

            let parser = new ExprParser(tokenStream);
            let errorListener = new ErrorListener();
            parser.addErrorListener(errorListener);
            let tree = parser.expression();
            expect(errorListener.errorCount, "Test 1").equals(0);

            let core = new c3.CodeCompletionCore(parser);

            // 1) At the input start.
            let candidates = core.collectCandidates(0);

            expect(candidates.tokens.size, "Test 2").to.equal(3);
            expect(candidates.tokens.has(ExprLexer.VAR), "Test 3").to.equal(true);
            expect(candidates.tokens.has(ExprLexer.LET), "Test 4").to.equal(true);
            expect(candidates.tokens.has(ExprLexer.ID), "Test 5").to.equal(true);

            expect(candidates.tokens.get(ExprLexer.VAR), "Test 6").to.eql([ExprLexer.ID, ExprLexer.EQUAL]);
            expect(candidates.tokens.get(ExprLexer.LET), "Test 7").to.eql([ExprLexer.ID, ExprLexer.EQUAL]);
            expect(candidates.tokens.get(ExprLexer.ID), "Test 8").to.eql([]);

Given the other errors I was seeing, I'm assuming there's something fundamentally different about the JavaScript parser/lexer than the ones generated by antlr4ts-cli and something is still missing. Any help or insight as to what might be different would be useful.

[mike-lischke]

@kaidjohnson You know you are writing to a closed issue? Please don't hijack already finished issues but create a new one, if required.

Indeed there are differences between the JS and TS runtime versions. The authors of @antlr4ts insist on implementing their runtime in a way that is optimal for the TS language, thereby sacrificing compatibility. Best would be you open an issue against @antlr4ts, asking for more compatibility. However, I remember a discussion where people spoke about making the TS target the main one and generating the JS target from it. The current handwritten JS target would then be obsolete.

[kaidjohnson]

@mike-lischke It wasn't my intention to "hijack" anything. The issue I was seeing was the same as the user that posted the original issue here, and I thought I might get some additional insight into what the options were to get around it. Every git repo owner seems to have a different preference for creating tickets -- some complain about opening duplicates, some complain about extending closed tickets. It's always a 50/50 gamble; didn't mean to ruffle any feathers.

My current JS target was not handwritten, but rather generated via the core antlr4 library.

FWIW, I did work around the issue after a bit more trial-and-error by generating my parser and lexer via antlr4ts, converting the resulting ts (es2015 target) to js (using babel) -- the typescript es5 target wasn't handling the class conversion well -- and then using the core antlr4 runtime (not the ts version) for use in the browser. The parser needed to patched slightly to be compatible with the core Parser class from antlr4ts, but after that small patch (see below), I am getting tokens back.

Here's the patch, which essentially just copies over the missing method from antlr4ts:

import BaseExprParser from './ExprParser.ts';

class ExprParser extends BaseExprParser {
	constructor(...args) {
		super(...args);

		// The antlr4ts Parser expects `tryLT` to be defined on the token stream
		// and `LT` to defer to the `tryLT`.
		// @see 'node_modules/antlr4ts/Parser.js
		// @see 'node_modules/antlr4ts/BufferedTokenStream.js:203
		this._input.tryLT = this._input.LT;
		this._input.LT = function (k) {
			const result = this.tryLT(k);
			if (result === undefined) {
				throw new RangeError('requested lookback index out of range');
			}
			return result;
		};
	}
}

For now, this resolves the issue for me and I am able to run the antlr4-c3 engine in the browser. Should I run into any other issues, I'll be sure to open a new ticket. Thanks!