'CommonToken' object has no attribute 'stopIndex' in Python{2|3}

[KvanTTT]

Python code

Consider the following Python3 code and custom grammar Separated:

import sys;
from antlr4 import *
from SeparatedLexer import SeparatedLexer
from SeparatedParser import SeparatedParser

def main(argv):
    code = open('Text', 'r').read()
    codeStream = InputStream(code)
    lexer = SeparatedLexer(codeStream)
    tokens = lexer.getAllTokens()
    tokensSource = ListTokenSource.ListTokenSource(tokens)
    tokensStream = CommonTokenStream(tokensSource)
    #tokensStream = CommonTokenStream(lexer) # use this statement to overcome the bug
    parser = SeparatedParser(tokensStream)
    tree = parser.rule1()
    print("Tree ", tree.toStringTree(recog=parser));

if __name__ == '__main__':
    main(sys.argv)

It's not working properly. The following error occurs:

Error

Traceback (most recent call last):
  File "main.py", line 20, in <module>
    main(sys.argv)
  File "main.py", line 16, in main
    tree = parser.rule1()
  File "SeparatedParser.py", line 122, i
n rule1
    localctx.t1 = self.a()
  File "SeparatedParser.py", line 326, i
n a
    self.match(SeparatedParser.A)
  File "Parser.py", line 113, in match
    self.consume()
  File "Parser.py", line 329, in consume
    self.getInputStream().consume()
  File "BufferedTokenStream.py", line 99, in consume
    if self.sync(self.index + 1):
  File "BufferedTokenStream.py", line 111, in sync
    fetched = self.fetch(n)
  File "BufferedTokenStream.py", line 123, in fetch
    t = self.tokenSource.nextToken()
  File "ListTokenSource.py", line 81, in nextToken
    previousStop = self.tokens[len(self.tokens) - 1].stopIndex
AttributeError: 'CommonToken' object has no attribute 'stopIndex'

Java

Meanwhile the Java code works good:

String code = readFile(args[0]);
ANTLRInputStream codeStream = new ANTLRInputStream(code);
SeparatedLexer lexer = new SeparatedLexer(codeStream);
List<? extends Token> tokens = lexer.getAllTokens();
ListTokenSource tokensSource = new ListTokenSource(tokens);
CommonTokenStream tokensStream = new CommonTokenStream(tokensSource);
SeparatedParser parser = new SeparatedParser(tokensStream);
ParserRuleContext ast = parser.rule1();
String stringTree = ast.toStringTree(parser);
System.out.print("Tree " + stringTree);

So, I am not able to parse the custom tokens list.

[KvanTTT]

@parrt the label improvement should be replaced with bug. Also this issue can be closed for now.