bpo-40939: Add the new grammar to the grammar specification documentation #19969

pablogsal · 2020-05-06T21:00:16Z

Example of how it looks like when using the new lexer:

https://bugs.python.org/issue40939

gvanrossum

I worry about some of the rather verbose comments in the grammar file, and the various error rules. Also there are some cases where lookaheads are essential (notably the way formal parameters are now parsed). And maybe the ordering and formatting of some of the rules could be better.

PS. Your initial comments still shows variables (a=...) but the script correctly erases those.

Doc/tools/extensions/peg_highlight.py

pablogsal · 2020-05-06T21:35:20Z

I worry about some of the rather verbose comments in the grammar file, and the various error rules.

I can try to eliminate alternatives if they start with "invalid_" but I think it will not be super resilient :(

Also there are some cases where lookaheads are essential (notably the way formal parameters are now parsed).

Would you like to bring lookaheads back?

And maybe the ordering and formatting of some of the rules could be better.

Not much we can do here with a custom lexer unless we touch the original file.

PS. Your initial comments still shows variables (a=...) but the script correctly erases those.

Yeah, I removed the variables in the last amend after I realized that we wanted to get rid of them.

gvanrossum · 2020-05-06T22:43:47Z

I worry about some of the rather verbose comments in the grammar file, and the various error rules.

I can try to eliminate alternatives if they start with "invalid_" but I think it will not be super resilient :(

Plus, you'd have to eliminate both the alternatives and the rules. But these really do not deserve to be listed in the reference manual.

Also there are some cases where lookaheads are essential (notably the way formal parameters are now parsed).

Would you like to bring lookaheads back?

Only the ones that are significant -- and those in formal parameters are significant (without them, def foo(bar baz) would be legal)

And maybe the ordering and formatting of some of the rules could be better.

Not much we can do here with a custom lexer unless we touch the original file.

Yeah, we can do that later.

pablogsal · 2020-05-06T22:46:50Z

Only the ones that are significant -- and those in formal parameters are significant (without them, def foo(bar baz) would be legal)

Hmmm.... is there any obvious automatic way (using regular expressions) to detect the ones we want from the ones we don't want? Otherwise, we may need to maintain a copy of the grammar for displaying here and manually trim what we need.

gvanrossum · 2020-05-06T22:53:29Z

I dunno. Let's think about this some more before rushing into it. (We'd have to explain a big about PEG's ordering of alternatives too, to guide people using this as a basis for anything.)

lysnikolaou · 2020-05-06T23:05:51Z

Hmmm.... is there any obvious automatic way (using regular expressions) to detect the ones we want from the ones we don't want?

A heuristic that is currently 100% correct is that a lookahead is only used for optimisation and can be skipped if and only if it is a positive lookahead and it appears first in its alternative. Although this is currently correct, it seems like something that could very easily be invalidated in the future, so I don't know if it can be used in the script.

willingc

Overall, I like the approach. There are a few places where color makes a line choppy and hard to follow and may make it more difficult to read than just black text.

…tion

Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>

gvanrossum · 2020-07-26T22:00:06Z

I think I'm going to do something with this after all. Let's see if we can't make the highlighter also remove invalid_* rules and comments.

gvanrossum · 2020-07-26T22:41:20Z

Okay, @lysnikolaou we may need your review because now it's both Pablo's and my code.

pablogsal · 2020-07-27T10:45:08Z

Should we backport this to 3.9? (Minus the Grammar/Grammar delete. :-)

I think it makes sense to do the backport. What do you think @lysnikolaou ?

gvanrossum · 2020-07-27T14:25:56Z

Okay then @pablogsal what do you think of my version?

willingc

LGTM. Thanks @pablogsal and @gvanrossum.

pablogsal

Okay then @pablogsal what do you think of my version?

I think is great :) I left a minor question, but feel free to ignore

Note: I cannot approve my own PR, so go ahead and merge!

Doc/reference/grammar.rst

Doc/tools/extensions/peg_highlight.py

lysnikolaou · 2020-07-27T15:10:17Z

Should we backport this to 3.9? (Minus the Grammar/Grammar delete. :-)

I think it makes sense to do the backport. What do you think @lysnikolaou ?

I also think that backporting this to 3.9 makes sense.

gvanrossum · 2020-07-27T17:25:28Z

The new version I pushed has a longer intro in grammar.rst, both to explain (some of) what's omitted and to clarify the notation. It also restores most lookaheads, except for the positive ones that are purely optimizations (as a heuristic I use | followed by a positive lookahead). It makes the grammar a little uglier but reduces the possibility of looking like an ambiguity.

lysnikolaou

LGTM. Thanks!!

miss-islington · 2020-07-27T18:20:40Z

Thanks @pablogsal for the PR, and @gvanrossum for merging it 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

miss-islington · 2020-07-27T18:20:46Z

Sorry, @pablogsal and @gvanrossum, I could not cleanly backport this to 3.9 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 72cabb2aa636272e608285f5a6ba83b62be9be4e 3.9

…cumentation (GH-19969) (We censor the heck out of actions and some other stuff using a custom "highlighter".) Co-authored-by: Guido van Rossum <guido@python.org>. (cherry picked from commit 72cabb2) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>

bedevere-bot · 2020-07-27T18:31:36Z

GH-21641 is a backport of this pull request to the 3.9 branch.

…cumentation (GH-19969) (#21641) (We censor the heck out of actions and some other stuff using a custom "highlighter".) (cherry picked from commit 72cabb2) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>

…ation (pythonGH-19969) (We censor the heck out of actions and some other stuff using a custom "highlighter".) Co-authored-by: Guido van Rossum <guido@python.org>

pablogsal added the skip issue label May 6, 2020

pablogsal requested review from gvanrossum and lysnikolaou May 6, 2020 21:00

the-knights-who-say-ni added the CLA signed label May 6, 2020

bedevere-bot added the awaiting core review label May 6, 2020

pablogsal self-assigned this May 6, 2020

pablogsal requested a review from willingc May 6, 2020 21:01

pablogsal added skip news and removed skip issue labels May 6, 2020

pablogsal force-pushed the grammar branch from 806b72f to 360b195 Compare May 6, 2020 21:05

gvanrossum reviewed May 6, 2020

View reviewed changes

Doc/tools/extensions/peg_highlight.py Outdated Show resolved Hide resolved

willingc reviewed May 7, 2020

View reviewed changes

lysnikolaou mentioned this pull request May 7, 2020

Refactor Grammar for publishing on Python Docs we-like-parsers/pegen_experiments#69

Closed

pablogsal mentioned this pull request May 11, 2020

Labeled Exceptions for pegen - Better error-reporting in CPython we-like-parsers/cpython#123

Closed

pablogsal mentioned this pull request May 26, 2020

Fix lookahead of soft keywords in the PEG parser #20436

Merged

pablogsal mentioned this pull request Jun 11, 2020

Document the new grammar we-like-parsers/cpython#135

Closed

gvanrossum changed the title ~~bpo-40334: Add the new grammar to the grammar specification documentation~~ bpo-40939: Add the new grammar to the grammar specification documentation Jul 26, 2020

pablogsal and others added 2 commits July 26, 2020 14:55

bpo-40334: Add the new grammar to the grammar specification documenta…

de4d919

…tion

Update Doc/tools/extensions/peg_highlight.py

22e6c71

Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>

gvanrossum force-pushed the grammar branch from 2f674b7 to 22e6c71 Compare July 26, 2020 21:57

gvanrossum added 2 commits July 26, 2020 15:10

Tweak text intro to the grammar

c83284e

Suppress all rules named invalid_* or incorrect_*

27f7f00

Now we can remove Grammar/Grammar

266a99a

willingc approved these changes Jul 27, 2020

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Jul 27, 2020

pablogsal commented Jul 27, 2020

View reviewed changes

Doc/reference/grammar.rst Show resolved Hide resolved

pablogsal commented Jul 27, 2020

View reviewed changes

Doc/tools/extensions/peg_highlight.py Show resolved Hide resolved

lysnikolaou reviewed Jul 27, 2020

View reviewed changes

Doc/tools/extensions/peg_highlight.py Show resolved Hide resolved

gvanrossum added 2 commits July 27, 2020 10:20

More conservative suppression of lookaheads

13aa70d

Lengthen intro for grammar.rst

7705e02

Fix typo and comma

bac8971

gvanrossum added the needs backport to 3.9 only security fixes label Jul 27, 2020

lysnikolaou approved these changes Jul 27, 2020

View reviewed changes

gvanrossum merged commit 72cabb2 into python:master Jul 27, 2020

bedevere-bot removed the awaiting merge label Jul 27, 2020

miss-islington assigned gvanrossum and unassigned pablogsal Jul 27, 2020

bedevere-bot removed the needs backport to 3.9 only security fixes label Jul 27, 2020

pablogsal deleted the grammar branch May 19, 2021 18:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-40939: Add the new grammar to the grammar specification documentation #19969

bpo-40939: Add the new grammar to the grammar specification documentation #19969

pablogsal commented May 6, 2020 •

edited by gvanrossum

gvanrossum left a comment

pablogsal commented May 6, 2020 •

edited

gvanrossum commented May 6, 2020

pablogsal commented May 6, 2020

gvanrossum commented May 6, 2020 via email

lysnikolaou commented May 6, 2020

willingc left a comment

gvanrossum commented Jul 26, 2020

gvanrossum commented Jul 26, 2020

pablogsal commented Jul 27, 2020

gvanrossum commented Jul 27, 2020

willingc left a comment

pablogsal left a comment

lysnikolaou commented Jul 27, 2020

gvanrossum commented Jul 27, 2020

lysnikolaou left a comment

miss-islington commented Jul 27, 2020

miss-islington commented Jul 27, 2020

bedevere-bot commented Jul 27, 2020

bpo-40939: Add the new grammar to the grammar specification documentation #19969

bpo-40939: Add the new grammar to the grammar specification documentation #19969

Conversation

pablogsal commented May 6, 2020 • edited by gvanrossum

gvanrossum left a comment

Choose a reason for hiding this comment

pablogsal commented May 6, 2020 • edited

gvanrossum commented May 6, 2020

pablogsal commented May 6, 2020

gvanrossum commented May 6, 2020 via email

lysnikolaou commented May 6, 2020

willingc left a comment

Choose a reason for hiding this comment

gvanrossum commented Jul 26, 2020

gvanrossum commented Jul 26, 2020

pablogsal commented Jul 27, 2020

gvanrossum commented Jul 27, 2020

willingc left a comment

Choose a reason for hiding this comment

pablogsal left a comment

Choose a reason for hiding this comment

lysnikolaou commented Jul 27, 2020

gvanrossum commented Jul 27, 2020

lysnikolaou left a comment

Choose a reason for hiding this comment

miss-islington commented Jul 27, 2020

miss-islington commented Jul 27, 2020

bedevere-bot commented Jul 27, 2020

pablogsal commented May 6, 2020 •

edited by gvanrossum

pablogsal commented May 6, 2020 •

edited