GetAllTokens and ClearDFA sometimes throws NRE in multithread mode

[KvanTTT]

Lexing (and parsing) and cache clearing can not be used simultaniously due to a such multithread issue:

System.NullReferenceException: Object reference not set to an instance of an object.
   at Antlr4.Runtime.Dfa.DFAState..ctor(DFA dfa, ATNConfigSet configs)
   at Antlr4.Runtime.Atn.LexerATNSimulator.AddDFAState(ATNConfigSet configs)
   at Antlr4.Runtime.Atn.LexerATNSimulator.MatchATN(ICharStream input)
   at Antlr4.Runtime.Atn.LexerATNSimulator.Match(ICharStream input, Int32 mode)
   at Antlr4.Runtime.Lexer.NextToken()
   at AI.PM.PhpAstConversion.PHPLexer.NextToken() in 
   at Antlr4.Runtime.Lexer.GetAllTokens()

At present time it can be solved with ReadWriterLockSlim. See my publication for detail: memory-consumption-of-antlr-runtime.

[sharwell]

The ClearDFA method is not safe for use while parsing operations are in progress. This is a global restriction for a particular lexer or parser (i.e. there can be no FooParser parsing operations in progress for any instance of FooParser when ClearDFA is called on the interpreter for any instance of FooParser).

I believe this is more of a shortcoming in the documentation of a method that is generally discouraged than a bug in the implementation of the method.

[KvanTTT]

In multithread mode ReadWriterLockSlim can be used as it pointed out in my publication. It also can be added to a documentation.

[sharwell]

@KvanTTT It's not possible to introduce a lock on the hot path in prediction. Doing so would have devastating performance implications for known users running ANTLR 4 in highly concurrent environments. For now I'm documenting that ClearDFA is not safe for concurrent use with the explicit additional note that the ATN instance is shared by default across recognizer instances (Lexer and Parser).

The best way to manage the DFA size in a concurrent environment is to explicitly set the ATN instance associated with each recognizer instance. The following code references an ATN instance in _currentATN. Rather than directly constructing MyLexer, call MyLexerFactory.CreateLexer. And rather than directly clearing the DFA, call MyLexerFactory.ClearDFA.

private ATN _currentATN;

public MyLexerFactory()
{
  // Initialize _currentATN
  ClearDFA();
}

public MyLexer CreateLexer(ICharStream input)
{
  MyLexer lexer = new MyLexer(input);
  lexer.Interpreter = new LexerATNSimulator(lexer, _currentATN);
  return lexer;
}

// Clear the DFA by forcing future instances of MyLexer to be created with a
// different ATN instance, which is initially cleared. Existing instances of
// MyLexer are not affected.
public void ClearDFA()
{
  _currentATN = new ATNDeserializer().Deserialize(MyLexer._serializedATN.ToCharArray());
}