Draft: Refactor PythonConsoleLexer as a DelegatingLexer

[jeanas]

This is simpler and more reliable than hand-coding the state machine.

Fixes #2411

I need to split this into more commits and check a few things before it can be 'undrafted'.

Also, while this should IMHO be the way going forward, I'm not sure about including a full rewrite of a lexer in a bugfix release. On the other hand, #2411 is not a regression compared to 2.14 as far as I can see.

[Anteru]

I like that solution better than the current workarounds. I just checked, the new lexer doesn't appear in the documentation, so that's good. I don't mind shipping a lexer rewrite given it passes all existing tests, and the current test coverage is better than what we shipped the last major update with (hence the regressions!) Unless @birkenfeld has any reservations, my vote is on merging this and getting a 2.15.1 release out of the door as soon as we can.

[birkenfeld]

I'm on holiday, my only reservations are in hotels. Go ahead :)

[Anteru]

@jeanas Good to go or do you want to do anything else before merging? It's currently in draft so I cannot trivially merge this.

[jeanas]

Coming back to this, I didn't really find value in making more commits. Let's merge it.

I agree on releasing a bugfix version soon. Maybe also include #2404 in it? (It fixes a case of catastrophic backtracking.)

[Anteru]

Thanks! Yes -- that's enough for a point release. I'll get #2404 included as well. I should get to it tomorrow. Good work on both!