Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide fault-tolerant parsing #3303

Open
fabioz opened this issue Jan 6, 2020 · 4 comments
Open

Provide fault-tolerant parsing #3303

fabioz opened this issue Jan 6, 2020 · 4 comments

Comments

@fabioz
Copy link

@fabioz fabioz commented Jan 6, 2020

As a note, I'm volunteering to implement that, so, this issue is mostly to know if this is a suitable addition to cython.

My use case is that I'm using cython itself to create an AST for users in an IDE (so, it may be requested while the code is not complete).

Right now I'm using the code below to generate an AST from cython:

from Cython.Compiler.TreeFragment import parse_from_strings
mod = parse_from_strings(name, source)

My idea would be adding an additional parameter to parse_from_strings (something as fault_tolerant=True -- with a default of False) and when an error is found in a statement ignore errors until a next valid statement is found.

Errors would be attached to the returned AST.

@fabioz

This comment has been minimized.

Copy link
Author

@fabioz fabioz commented Jan 10, 2020

I did some experiments here and just changing the fatal value in PyrexScanner.error (i.e.: https://github.com/cython/cython/blob/master/Cython/Compiler/Scanning.py#L480) to False (and storing the exceptions to be retrieved later) seems to be enough to make the parser fault-tolerant.

I was going to just monkey-patch this, but because those files are cython-compiled in the distribution, this is not possible (so, would it be ok if I did provide a pull request implementing this?)

@scoder

This comment has been minimized.

Copy link
Contributor

@scoder scoder commented Jan 11, 2020

Parser errors are usually fatal for the compiler, with only a few exceptions, but I wouldn't mind having a scanner-local setting that disables the raise in PyrexScanner.error(). Note that the attribute needs to be added to the Scanning.pxd as well. I think it's enough to add that option to the TreeFragment API for now, if that suits your use case. It can be added to other interfaces of the parser later, if necessary.

PR welcome.

@scoder

This comment has been minimized.

Copy link
Contributor

@scoder scoder commented Jan 11, 2020

Note that there are probably code sections in the parser code that implicitly expect an error() to raise and exit, instead of continuing with the rest of the code. If you run into such code, it should usually not be too hard to fix. PRs welcome for that, too.

@fabioz

This comment has been minimized.

Copy link
Author

@fabioz fabioz commented Jan 11, 2020

Great, I'll take a look into that then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.