Lexer default instance is set in class before keywords are added

[gshank]

In https://github.com/dbt-labs/dbt-core we use sqlparse to merge ctes at a particular place in our code. Multiple threads are started at the same time, and because the Lexer default instance is set before the keywords are added we were getting intermittent failures because we were parsing with a Lexer with no keywords.

In order to avoid this problem we had to initialize the Lexer earlier in the process: https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/compilation.py#L527

The keywords should be added to the Lexer instance before it is is stored in the class.

[andialbrecht]

Agreed. The Lexer shouldn't change iff it's a singleton and it should be thread-safe to use.