Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow PLY to generate parsing/lexing tables. #63

Merged
merged 1 commit into from Nov 24, 2019

Conversation

@FLHerne
Copy link
Contributor

FLHerne commented Nov 21, 2019

...and modify setup.py to create them at build time.

On my laptop, pre-generated tables reduce the time spent in
NMLParser.__init__() from 2.2s to 0.2s, halving the total runtime
of nmlc for small input files.

PLY will attempt to regenerate these automatically at runtime if the
grammar has changed. In the case where they're system-installed and not
writeable, the grammar is also system-installed and not writeable. :P

@glx22
Copy link
Contributor

glx22 commented Nov 21, 2019

A small change is required for pyinstaller. I made the changes locally and can push them to your branch if it's ok.

@FLHerne
Copy link
Contributor Author

FLHerne commented Nov 21, 2019

@glx22 Yes, please go ahead. :-)

@andythenorth
Copy link
Contributor

andythenorth commented Nov 21, 2019

I tested this with Iron Horse and --verbosity=4 flag. Python is 3.5.

Results:

  • Init parser: consistently 0.0s with this PR, 0.4-0.5s in master
  • Parsing: less consistent results, likely due to other system load. With this PR 16.4-17.3s. Master is 17.0-20.0s.
@glx22
Copy link
Contributor

glx22 commented Nov 22, 2019

Hmm pip install -e . or python setup.py develop fails with the setup.py changes if ply is not yet installed. Without the changes the dependencies are installed and everything is fine.

Copy link
Contributor

glx22 left a comment

The suggested changes fix the develop issue

setup.py Outdated Show resolved Hide resolved
setup.py Show resolved Hide resolved
@FLHerne FLHerne force-pushed the FLHerne:gen-ply-tabfiles branch from cd37ef9 to 1caaa30 Nov 22, 2019
...and modify setup.py to create them at build time.

On my laptop, pre-generated tables reduce the time spent in
 `NMLParser.__init__()` from 2.2s to 0.2s, halving the total runtime
 of nmlc for small input files.

PLY will attempt to regenerate these automatically at runtime if the
 grammar has changed. In the case where they're system-installed and not
 writeable, the grammar is also system-installed and not writeable. :P

This patch includes changes suggested by glx22.
@FLHerne FLHerne force-pushed the FLHerne:gen-ply-tabfiles branch from 1caaa30 to 02877cf Nov 24, 2019
@LordAro LordAro merged commit 09e8bf8 into OpenTTD:master Nov 24, 2019
12 checks passed
12 checks passed
Python 3.5 on ubuntu-latest
Details
Python 3.6 on ubuntu-latest
Details
Python 3.7 on ubuntu-latest
Details
Python 3.8 on ubuntu-latest
Details
Python 3.5 on macOS-latest
Details
Python 3.6 on macOS-latest
Details
Python 3.7 on macOS-latest
Details
Python 3.8 on macOS-latest
Details
Python 3.5 on windows-2016
Details
Python 3.6 on windows-2016
Details
Python 3.7 on windows-2016
Details
Python 3.8 on windows-2016
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.