@@ -17,7 +17,7 @@ core library is not specific to text and will work over any iterable type. parsers. * `examples\xml.py` is an example implementation of a parser for a reasonable subset of xml. - * `examples\calculatpr.py` is an example implementation of infix arithmetic + * `examples\calculator.py` is an example implementation of infix arithmetic parser and evaluator. * `test.py` runs all the test cases found in `tests` @@ -48,6 +48,74 @@ and to make a parser that accepts many 'a's: many_as = partial(many, a) +## How do I… + +Heres is a glossary of parsers and combinators by outcome to help learn to use Picoparse. Look at doc comments for specifics. + +### Match input + +Matching input is done by calling primitive parsers. + + * `picoparse.one_of` Match one item in the input stream that is equal to the + argument. Requires that the item supports `==`. + * `picoparse.not_one_of` Match one item that is not equal to the argument. + * `picoparse.satisfies` Matches one item that satisfies a guard function. + * `picoparse.any_token` Matches one of any item. + * `picoparse.eof` Matches the end of input. + * `picoparse.text.whitespace_char` Matches a single whitespace character + * `picoparse.text.newline` Matches a single newline character + * `picoparse.text.quote` Match one single or double quote + +### Match multiple items + +Matching multiple items is achieved by passing a primitive parser to a combinator function. These parsers are all greedy + + * `picoparse.many` Matches a parser zero or more times. + * `picoparse.many1` Matches a parser one or more times. + * `picoparse.many_until` Matches a parser zero or more times until the + terminating parser matches. + * `picoparse.many_until1` Matches a parser one or more times until the + terminating parser matches. + * `picoparse.sep` Matches a parser zero or more times, with a separator being + matched between each pair. + * `picoparse.sep` Matches a parser one or more times, with a separator being + matched between each pair. + * `picoparse.optional` Matches zero or one of parser. + * `picoparse.text.whitespace` Match zero or more whitespace characters + * `picoparse.text.whitespace1` Match one or more whitespace characters + +### Make a choice + +Choosing between possible parsers is achieved with the `choice` combinator, often assisted by `tri`. + + * `picoparse.choice` Match one of the parsers passed in order. If none match, + the choice fails to match as well. + * `picoparse.tri` should decorate a parser that consumes multiple pieces of + input. You can `picoparse.commit` to the choice if you know that no + other choice should succeed. (see calculator and xml examples for this in + use.) `tri` will automatically commit if it reaches the end of the decorated + parser. + +### Match a sequence + + * `picoparse.string` will apply `one_of` in for each item in the iterable + argument. Note that this is any string of matches, not just for character + parsing. + * `picoparse.text.caseless_string` will match the given str without regard + for upper or lower case (text only) + * `picoparse.cue` for something ignorable that cues you to what you really + want to match. eg, matching "#x" in a hex parser + * `picoparse.follow` for something ignorable that follows what you really want + to match. eg, matching "l" after a long integer. + * `picoparse.seq` for matching a specific set of (possibly named) parsers. + +### Matching something wrapped + + * `picoparse.text.lexeme` match a parser with optional whitespace on either + side. + * `picoparse.text.quoted` Match a parser with in single or double quote + characters until a matching quote is found. + ## Tracking development and reporting bugs This project is tracked with GIT via [GitHub](http://github.com/brehaut/picoparse/), and uses README.markdown @@ -27,11 +27,13 @@ for instance the quote and quoted parsers assume quotes can be ' or " # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE # POSSIBILITY OF SUCH DAMAGE. +from string import whitespace as _whitespace_chars + from picoparse import string, one_of, many, many1, many_until, any_token, run_parser from functools import partial quote = partial(one_of, "\"'") -whitespace_char = partial(one_of, ' \t\n\r') +whitespace_char = partial(one_of, _whitespace_chars) whitespace = partial(many, whitespace_char) whitespace1 = partial(many1, whitespace_char) newline = partial(one_of, '\n') picoparse/text.py @@ -1,4 +1,29 @@ #!/usr/bin/env python + +# Copyright (c) 2009, Andrew Brehaut, Steven Ashley +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# +# * Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE +# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + import unittest from tests import * test.py @@ -1,2 +1,28 @@ -from core_parsers import * + +# Copyright (c) 2009, Andrew Brehaut, Steven Ashley +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# +# * Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE +# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + from backend import * +from core_parsers import * +from text import * \ No newline at end of file tests/__init__.py @@ -1,3 +1,27 @@ +# Copyright (c) 2009, Andrew Brehaut, Steven Ashley +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# +# * Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE +# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + import unittest from picoparse import NoMatch, DefaultDiagnostics, BufferWalker @@ -118,4 +142,7 @@ class TestBufferWalker(unittest.TestCase): if __name__ == '__main__': unittest.main() -__all__ = ['TestDefaultDiagnostics', 'TestBufferWalker', ] +__all__ = [cls.__name__ for name, cls in locals().items() + if isinstance(cls, type) + and name.startswith('Test')] + tests/backend.py @@ -1,3 +1,27 @@ +# Copyright (c) 2009, Andrew Brehaut, Steven Ashley +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# +# * Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE +# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + import unittest from functools import partial as p @@ -267,9 +291,7 @@ class TestFuture(ParserTestCase): if __name__ == '__main__': unittest.main() -__all__ = ['TestTokenConsumers', 'TestManyCombinators', - 'TestSeperatorCombinators', 'TestSequencingCombinators', - 'TestFuture' - ] - +__all__ = [cls.__name__ for name, cls in locals().items() + if isinstance(cls, type) + and name.startswith('Test')] tests/core_parsers.py @@ -1,3 +1,27 @@ +# Copyright (c) 2009, Andrew Brehaut, Steven Ashley +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# +# * Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE +# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + import unittest from functools import partial as p from picoparse import run_parser, NoMatch tests/utils.py c9fecc84ad592633cfb566c61a101e1c0dd6229c Andrew Brehaut andrew@brehaut.net http://github.com/brehaut/picoparse/commit/2850cc2b84d7999f319c9945499800fdadca3e48 2850cc2b84d7999f319c9945499800fdadca3e48 2009-03-03T18:51:11-08:00 2009-03-03T18:51:11-08:00 readme has 'how do i…' section tests have license boilerplate and text module testing has begun a7e5a1fcd5a65c2c4677db68b550ab9ea77715b8 Andrew Brehaut andrew@brehaut.net