Replacing pyparsing with shlex

[tleonhardt]

Python's built-in "shlex" module for simple lexical analysis is probably good enough to handle our command-line parsing in cmd2 and it is implemented in C, so its performance would probably be a lot better than pyparsing, which is implemented in pure Python.

ANTLR and PLY also look like potentially superior alternatives to pyparsing, which is slow, dated, and at least in our case requires an overly-complicated grammar.

But shlex should be preferred since it is built-in to Python and not 3rd-party.

shlex:
https://docs.python.org/3.6/library/shlex.html

pyparsing:
http://pyparsing.wikispaces.com

ANTLR:
https://github.com/antlr/antlr4

PLY:
https://github.com/dabeaz/ply

This is a long-term "down the road" idea.

[tleonhardt]

@FedericoCeratto
I've researched shlex a little bit and it looks like it could serve as a a viable replacement for pyparsing.

The advantages of using shlex over pyparsing would be:

• Reduce dependency on 3rd-party modules which aren't built into the Python standard library
• Likely a significant performance boost in terms of CPU time to parse

But moving to shlex would necessitate a non-backwards compatible feature change in how we deal with comments. Currently both Python-style ("#...") and C-style ("/* ... */") comments are supported. If we switch from pyparsing to shlex, then we would only be able to support Python-style comments. The usage of comments really only makes sense in scripts.

In your opinion, would this minor backwards incompatibility be acceptable?

[tleonhardt]

shlex is just a lexer, not a full parser. So shlex is capable of doing a fine job of tokenizing the input. But we would still need to write custom logic to properly parse the tokens.

[kotfu]

I have no experience with any of the parsing tools linked above, but in 30 minutes of trying to learn a bit about each one, I think PLY seems like the best fit for cmd2. ANTLR doesn't feel very python-ish, given that you use a java program to define the grammar and then have it auto-generate python code for you. shlex only tokenizes and would require us to write a custom parser. pyparsing is slow (for us, and maybe for others too). ply has a pythonesque interface, is well maintained, and doesn't require any C extensions.

[tleonhardt]

I think either PLY or shlex would be reasonable options, with PLY probably being the path of least resistance.

Regardless this issue will be somewhat of a pain to work on because it will involve two learning curves:

• First figuring out how pyparsing works along with how the current cmd2 pyparsing-based parsing implementation functions
• Second learning how PLY works and how to re-factor the current cmd2 parsing to use PLY instead of pyparsing

On the other, hand, the performance benefits will likely be substantial - a factor of 5x or better is likely. These performance improvements probably won't be very noticeable during pure human interaction, but for scripts and/or unit tests they will likely be dramatic.

[kotfu]

Created new branch ply for this work.

[kotfu]

Tried implementing it with ply, spent lots of time and got frustrated trying to make the pipe work.

Tried implementing it with shlex, and made rapid progress. I think I'm gonna switch horses and try and finish it just using shlex.