Permalink
Browse files

Some reworks to pynetree.Node now providing a dump() function, and re…

…moving this function from pynetree.Parser. Furthermore, pynetree.Parser.parse won't return a list of nodes now, but always a pynetree.Node() on parse success.
  • Loading branch information...
phorward committed Jul 25, 2017
1 parent fe59b24 commit 9d3dffb11e5052b87ea5e2a6bd58a34e8d127497
Showing with 86 additions and 181 deletions.
  1. +45 −119 README.md
  2. +1 −1 examples/calc.py
  3. +1 −1 examples/calcbnf.py
  4. +1 −1 examples/demo.py
  5. +1 −1 examples/first.py
  6. +1 −1 examples/firstbnf.py
  7. +2 −2 examples/xpl.py
  8. +34 −55 pynetree.py
View
164 README.md
@@ -2,18 +2,13 @@
![Image of a Tree](pine.jpg)
# DESCRIPTION #
# About
**pynetree** is a simple, light-weight parsing toolkit for Python.
**pynetree** is both a Python library and a command-line utility for parsing.
The toolkit is primarily a library that provides an interface for defining,
running and processing parsers through a consistent and easy-to-use interface.
Parsing is the process of transferring input matching a particular grammar, like e.g. a program source code, into a well-formed data structure. This data structure is a so called abstract syntax tree (AST). pynetree is a tool doing all this for you: It takes a grammar description, runs a parser on provided input and generates the abstract syntax tree on a successful parse. This AST can than be taken to perform subsequent tasks, like writing an interpreter, compiler or any other kind of software requiring a parser.
The result of a parser run is an abstract syntax tree that is constructed during
the parse and can be used for further traversal or analyzation.
The following example program defines a simple expressional language, runs a
parser on it and prints the generated abstract syntax tree.
The following Python example using pynetree defines a simple two-function calculator as an expression language, runs a parser on it, and dumps out the generated abstract syntax tree.
from pynetree import Parser
@@ -30,9 +25,9 @@ parser on it and prints the generated abstract syntax tree.
expr$: add | term;
""")
p.dump(p.parse("1 + 2 * ( 3 + 4 ) * 5"))
When this program is ran from a console, a proper abstract syntax tree will
p.parse("1 + 2 * ( 3 + 4 ) * 5").dump()
When this program is run from a console, a proper abstract syntax tree will
be generated and printed, which shows the hierarchical structure of the parsed
expression.
@@ -46,56 +41,39 @@ expression.
int (4)
int (5)
Grammars and parsers can also be rapidly prototyped using pynetree's build-in
command-line utility. The next command-line call yields in exactly the same
abstract syntax trees, although some symbol names where shortened and both
grammar and input are expressed in a more compact way.
pynetree also provides a handy command-line tool to rapidly prototype grammars. The next command just generates the same parser as the example program from above.
$ ./pynetree.py "@int /[0-9]+/; f: int | '(' e ')'; t: @mul( t '*' f ) | f; e: @add( e '+' t ) | t;"
The pynetree project is currently under heavy development, so that changes in
API, function names, syntax or semantics may occur and need existing projects
to be ported.
# Requirements
It has been developed in the course of implementing a top-down parser supporting
left-recursive grammars. Therefore, pynetree is a parser that implements a
modified version of the packrat parsing algorithm, but with the approach to
provide true BNF-styled grammars, as known from other parser development tools
and frameworks.
pynetree is written in pure Python. It runs natively with any Python >= 2.5.
Have fun!
The only import done so far is the build-in `re` module for regular expression support. Nothing more is required!
# Features
## FEATURES ##
pynetree so far provides
The pynetree parser development toolkit so far provides
- A top-down packrat parser with support of direct and indirect
left recursive grammars.
- Mostly linear parsing time, even for left-recursive grammars.
- A top-down packrat parser with support of direct and indirect left recursive grammars.
- Mostly linear parsing time, even for left-recursive grammars due a powerful memorization algorithm.
- Grammars can be expressed as dict objects or by a BNF-grammar.
- Support functions for generating and traversing abstract syntax trees (AST).
- Lexical analysis can be performed via regular expressions (re), string or
by Python callables.
Please check out http://pynetree.org to get help and the newest updates on
the pynetree project.
- Lexical analysis can be performed via regular expressions (re), string or by Python callables.
## REQUIREMENTS ##
Please check out http://pynetree.org to get help and the newest updates on the pynetree project.
pynetree is written in pure Python. It runs natively with Python 2.x and 3.x.
The pynetree project is under heavy development, so that changes in API, function names, syntax or semantics may occur and need existing projects to be ported.
The only import done so far is the build-in `re` module for regular expression
support. Nothing else is required!
It has been developed in the course of implementing a top-down parser supporting left-recursive grammars. Therefore, pynetree is a parser that implements a modified version of the packrat parsing algorithm, but with the approach to provide true BNF-styled grammars, as known from other parser development tools and frameworks.
# GETTING STARTED #
# Getting started
pynetree is not a parser generator in classic terms like yacc or bison. It
can be seen as a library to directly express and parse the desired grammar
within Python code.
## INVOCATION VIA COMMAND-LINE ##
## Command-line interface
Nevertheless, a command-line interface is provided for rapidly grammar
prototyping and testing.
@@ -117,63 +95,33 @@ prototyping and testing.
'grammar' and 'input' can be either supplied as strings or files.
## Using it as a library
## USING FROM PYTHON SOURCE ##
For using pynetree in a Python script, it simply is required to create an object of the class `pynetree.Parser`.
For using pynetee in a Python script, it simply is required to create an object
of the class `pynetree.Parser`.
The Parser class requires a BNF-grammar to describe the language that shall be parsed as parameter. This grammar can be expressed in two different ways:
The Parser class requires a BNF-grammar to describe the language that shall be
parsed as parameter. This grammar can be expressed in two different ways:
1. As dict specifying the non-terminals as keys and their left-hand sides as the values (where the left-hand sides can be provided as list of strings). This requires much more configuration on the parsers token and emitting behavior, but is the Python-specific way.
1. As dict specifying the non-terminals as keys and their left-hand sides
as the values (where the left-hand sides can be provided as list of
strings). This requires much more configuration on the parsers token and
emitting behavior, but is the Python-specific way.
2. The other, more flexible method for specifying a grammar is pynetree's own grammar definition language, which itself internally is implemented using pynetree.
2. The other, more flexible method for specifying a grammar is pynetree's own
grammar definition language, which itself internally is implemented using
pynetree.
This language is oriented on the classic BNF-style, but supports several syntactical elements to quickly define a language and its behavior for token definition and abstract syntax tree (AST) construction.
This language is oriented on the classic BNF-style, but supports several
syntactical elements to quickly define a language and its behavior for
token definition and abstract syntax tree (AST) construction.
After the grammar is generally defined, the parser can be fine-tuned or extended to various features.
After the grammar is generally defined, the parser can be fine-tuned or extended
to various features.
- `pynetree.Parser.token()` is used to define named terminal symbols, which can be regular expression patterns, static strings or callables.
- `pynetree.Parser.ignore()` is used for the definition of whitespace tokens, which are generally allowed between all other tokens.
- `pynetree.Parser.emit()` is used to define symbols (both non-terminal or terminal) that are emitted as nodes in AST. Terminal symbols will always define leafs in the AST, where non-terminals can emit leafs if no sub-ordered symbols are emitted. (In a full parse tree, non-terminals will never be leafs, but nodes).
- `pynetree.Parser.token()` is used to define named terminal symbols, which can
be regular expression patterns, static strings or callables.
- `pynetree.Parser.ignore()` is used for the definition of whitespace tokens,
which are generally allowed between all other tokens.
- `pynetree.Parser.emit()` is used to define symbols (both non-terminal or
terminal) that are emitted as nodes in AST. Terminal symbols will always
define leafs in the AST, where non-terminals can emit leafs if no sub-ordered
symbols are emitted. (In a full parse tree, non-terminals will never be leafs,
but nodes).
The final parsing of a string is performed by the function `Parser.parse()`. This function returns the AST for the parsed input. AST are consisting of `pynetree.Node` objects or - in case of a sequence of multiple elements in the same level - lists of `pynetree.Node` objects.
The final parsing of a string is performed by the function `Parser.parse()`.
To walk on such an AST, the function `pynetree.Node.dump()` can be used to print the AST in a well-formatted style, or `pynetree.Parser.traverse()` to possible call emitting functions on every node. It is also helpful to use these functions as templates for other, more specialized tree traversal and walker functions.
This function returns the AST for the parsed input. AST are consisting of
`pynetree.Node` objects or - in case of a sequence of multiple elements in the
same level - lists of `pynetree.Node` objects.
- `pynetree.Parser.parse()` parses an input string on the given grammar and returns and AST. The AST is represented as a hierarchy of `pynetree.Node` objects.
- `pynetree.Node.dump()` allows for dumping ASTs returned by `pynetree.Parser.parse()` in a well-formed style.
- `pynetree.Parser.traverse()` walks along an abstract syntax tree generated by `pynetree.Parser.parse()`, and performs function calls to perform top-down, pass-by and bottom-up tree traversal possibilities.
To walk on such an AST, the function `pynetree.Parser.dump()` can be used to
print the AST in a well-formatted style, or `pynetree.Parser.traverse()` to
possible call emitting functions on every node. It is also helpful to use these
functions as templates for other, more specialized tree traversal and walker
functions.
- `pynetree.Parser.parse()` parses an input string on the given grammar.
- `pynetree.Parser.dump()` dumps an abstract syntax tree generated by
`pynetree.Parser.parse()`.
- `pynetree.Parser.traverse()` walks along an abstract syntax tree generated by
`pynetree.Parser.parse()`, and performs function calls to perform top-down,
pass-by and bottom-up tree traversal possibilities.
When higher AST traversal features are required for a pynetree use-case, it is
recommended to sub-class the `pynetree.Parser`-class into an own class, which
could serve as some kind of compiler or interpreter, like here:
When higher AST traversal features are required for a pynetree parser, it is recommended to sub-class `pynetree.Parser` into a more specific class, serving as some kind of compiler or interpreter, like this example:
import pynetree
@@ -223,40 +171,18 @@ could serve as some kind of compiler or interpreter, like here:
c.traverse(c.parse("1337 - 42 + 23"))
Please do also take a look at the many examples provided with pynetree to get
familiar with these functions and possibilities.
# AUTHOR #
pynetree is developed and maintained by Jan Max Meyer and the company
Phorward Software Technologies.
Please do also take a look at the many examples provided with pynetree to get familiar with these functions and possibilities.
This project is one result of a several years experience in parser development
tools, and is currently worked out as some kind of sister-project of a library
called libphorward, that focuses on the C programming language. Therefore, the
BNF-styled grammar definition language of both pynetree and libphorward are
similar and should provide the same interface for both parser generation tools.
Take a look at https://bitbucket.org/codepilot/phorward for more information.
# Author
Help of any kind to extend and improve or enhance this project in any kind or
way is always appreciated.
pynetree is developed and maintained by Jan Max Meyer, Phorward Software Technologies.
pynetree is the result of a several years experience in parser development tools, and is currently worked out as some kind of sister-project of the (Phorward Toolkit/libphorward)[https://github.com/phorward/phorward], a parsing a lexical analysis toolkit implementing LR/LALR parsers and focusing on the C programming language. Therefore, the BNF-styled grammar definition language of both pynetree and libphorward are similar and provide the same interface for both parser development tools.
# LICENSING #
# Licensing
Copyright (C) 2015-2017 by Phorward Software Technologies, Jan Max Meyer.
Copyright (C) 2015-2017 by Jan Max Meyer, Phorward Software Technologies.
You may use, modify and distribute this software under the terms and conditions
of the MIT license. The full license terms are provided in the LICENSE file.
You may use, modify and distribute this software under the terms and conditions of the MIT license. The full license terms are provided in the LICENSE file.
THIS SOFTWARE IS PROVIDED BY JAN MAX MEYER (PHORWARD SOFTWARE TECHNOLOGIES) AS
IS AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL JAN MAX MEYER (PHORWARD SOFTWARE TECHNOLOGIES) BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
THIS SOFTWARE IS PROVIDED BY JAN MAX MEYER (PHORWARD SOFTWARE TECHNOLOGIES) AS IS AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL JAN MAX MEYER (PHORWARD SOFTWARE TECHNOLOGIES) BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
View
@@ -18,4 +18,4 @@
p.ignore(r"\s+") #ignore whitespace
p.token("INT", r"\d+")
p.dump(p.parse("1 + 2 * (3 + 4) + 5"))
p.parse("1 + 2 * (3 + 4) + 5").dump()
View
@@ -18,5 +18,5 @@
e$: add | sub | t;
""")
p.dump(p.parse("123 + 456 * 789"))
p.parse("123 + 456 * 789").dump()
View
@@ -65,7 +65,7 @@ def post_calc(self, node):
ast = calc.parse("1 + 2 * ( 3 + 4 ) * 5 - 6 / 7")
print("--- abstract syntax tree ---")
calc.dump(ast)
ast.dump()
# Traverse (interpret) the parse tree
print("--- traversal ---")
View
@@ -14,4 +14,4 @@
p.ignore(r"\s+")
p.token("INT", r"\d+")
p.dump(p.parse("1 + 2 * (3 + 4) + 5"))
p.parse("1 + 2 * (3 + 4) + 5").dump()
View
@@ -14,4 +14,4 @@
e: add | t;
""")
p.dump(p.parse("1 + 2 * (3+4) + 5"))
p.parse("1 + 2 * (3+4) + 5").dump()
View
@@ -85,7 +85,7 @@
# "99-Bottles-of-Beer" implemented in XPL:
p.dump(p.parse("""
p.parse("""
if( ( bottles = prompt( "Enter number of bottles [default=99]" ) ) == "" )
bottles = 99;
@@ -112,4 +112,4 @@
else
print( bottles + " more bottles of beer on the wall." );
}
"""))
""").dump()
Oops, something went wrong.

0 comments on commit 9d3dffb

Please sign in to comment.