-
-
Notifications
You must be signed in to change notification settings - Fork 276
Wikispaces Content
The previous website for pyparsing was hosted on the Wikispaces free wiki. This page replicates content archived from that site.
2011-07-23 08:21:11 - joslin01 - 'str' object is not callable
2010-05-03 04:22:51 - rgammans - (Easy_)Installing on windows....
2009-08-27 07:12:31 - akonsu - (newbie) setResultsName question
2009-08-31 23:05:00 - marjoj - (newbie) What is needed to run pyparsing?
2017-08-27 19:25:52 - vacaut - += operator and user function for math equation
2009-04-10 07:10:57 - JaimeWyant - 1.5.2 pypi release?
2015-10-19 05:12:31 - heronils - 2to3 required?
2009-04-02 13:41:59 - dbv - 64-bit Windows
2007-05-01 11:24:05 - shinewu - A bug in error msg
2006-08-28 12:17:57 - cie - a bug? combine and recursion
Hi, does anyone have any idea as to where to look for this kind of error message:
TypeError: 'str' object is not callable
The debug statements make it look like everything is going to get parsed perfectly, but then it hits this error and stops. Some of my grammars compile into classes that look like this:
class WhileStatement(CompoundStatement):
def __init__(self,t):
self.arg = t
pprint.pprint(t.asList())
def __str__(self):
return pprint.pformat(self.arg.asList())
They're not complete..
This is the entire traceback:
Traceback (most recent call last):
File 'tests.py', line 151, in \<module\>
test_files(arg + '.gup')
File 'tests.py', line 124, in test_files
tokens = grammar.file_input.parseFile(file)
File 'c:\python27\lib\site-packages\pyparsing.py', line 1410, in parseFile
return self.parseString(file_contents, parseAll)
File 'c:\python27\lib\site-packages\pyparsing.py', line 1021, in parseString
loc, tokens = self._parse( instring, 0 )
File 'c:\python27\lib\site-packages\pyparsing.py', line 894, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File 'c:\python27\lib\site-packages\pyparsing.py', line 2735, in parseImpl
loc, tokens = self.expr._parse( instring, loc, doActions, callPreParse=False
)
File 'c:\python27\lib\site-packages\pyparsing.py', line 894, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File 'c:\python27\lib\site-packages\pyparsing.py', line 2478, in parseImpl
ret = e._parse( instring, loc, doActions )
File 'c:\python27\lib\site-packages\pyparsing.py', line 894, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File 'c:\python27\lib\site-packages\pyparsing.py', line 2623, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File 'c:\python27\lib\site-packages\pyparsing.py', line 872, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File 'c:\python27\lib\site-packages\pyparsing.py', line 2435, in parseImpl
return maxMatchExp._parse( instring, loc, doActions )
File 'c:\python27\lib\site-packages\pyparsing.py', line 872, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File 'c:\python27\lib\site-packages\pyparsing.py', line 2478, in parseImpl
ret = e._parse( instring, loc, doActions )
File 'c:\python27\lib\site-packages\pyparsing.py', line 872, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File 'c:\python27\lib\site-packages\pyparsing.py', line 2368, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File 'c:\python27\lib\site-packages\pyparsing.py', line 872, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File 'c:\python27\lib\site-packages\pyparsing.py', line 2478, in parseImpl
ret = e._parse( instring, loc, doActions )
File 'c:\python27\lib\site-packages\pyparsing.py', line 872, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File 'c:\python27\lib\site-packages\pyparsing.py', line 2368, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File 'c:\python27\lib\site-packages\pyparsing.py', line 872, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File 'c:\python27\lib\site-packages\pyparsing.py', line 2769, in parseImpl
loc, tokens = self.expr._parse( instring, loc, doActions, callPreParse=False
)
File 'c:\python27\lib\site-packages\pyparsing.py', line 894, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File 'c:\python27\lib\site-packages\pyparsing.py', line 2623, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File 'c:\python27\lib\site-packages\pyparsing.py', line 907, in _parseNoCache
tokens = fn( instring, tokensStart, retTokens )
File 'c:\python27\lib\site-packages\pyparsing.py', line 675, in wrapper
return func(*args[limit[0]:])
I set a parse action as ('...') by accident!! So used to using setName('..') and ('...').
:) Glad you found your bug! This does seem to be a downside to the new parse action wrapping, in that exceptions raised in parse actions end up in some pretty cryptic stack traces. I'll try to look at that some more to see if I can be more selective about exception catching in the trim_arity code.
-- Paul
If pyparsing is a dependency of another then then setuptools can be configured to automatically call 'easy_install pyparsing' if it is not already installed.
Unfortunately it downloads the windows install and tries to run setup.py from inside it. - Which it doesn't contain.
See:-
C:\>c:\Python26\Scripts\easy_install.exe pyparsing
Searching for pyparsing
Reading
Reading
Reading
Reading
Reading
Reading
Best match: pyparsing 1.5.2
Downloading
1.5.2/pyparsing-1.5.2.win32.exe/download
Processing download
error: Couldn't find a setup script in c:\docume1\roger\locals1\temp\easy_inst
all-heexie\download
The executable is not easy to run either as it requires MSVCR71.dll which is not universally deployed on windows XP.
Is there any chance of having a window bdist zipfile or something similiar on pypi which will let this work.
Otherwise I hae to put a seriously complicated set of installation instructions for my app which the non-technical users will shy away from.
I have been promising this next release (1.5.3) for months now, but I think I'll actually get some time for it very soon, and I'll try to include a windows bdist zip as one of the options. As a workaround, you can just include the pyparsing.py file in with your own code. I intentionally kept pyparsing's code footprint down to a single source file, so it would be easy to include with a project's code.
If you download the pyparse zip file and then easy_install <location of zip file> it should work.
hello,
i have this grammar and a sample string:
from pyparsing import *
slug = Word(srange('[a-zA-Z0-9_-]'))
date = Keyword('date') + '=' + Combine(Word(nums, exact = 4) + ('-' + Word(nums, exact = 2)) * 2).setResultsName('date')
name = Keyword('name') + '=' + slug.setResultsName('name')
template = (Combine('.' + slug) + '=' + QuotedString('{', endQuoteChar = '}', multiline = True)).setResultsName('template', listAllMatches = True)
plate = Keyword('plate') + '=' + (date & name & ZeroOrMore(template)).setResultsName('plate', listAllMatches = True)
grammar = ZeroOrMore(plate) + StringEnd()
doc = '''
plate=
name= 1-my_Name
date= 2009-08-22
.summary= {
some long text
consisting of two lines
}
.default= {
even longer text
that can span even more
than two lines
}
'''
tokens = grammar.parseString(doc)
for t in tokens.plate[0].template :
print t
it procudes this output:
['.summary', '=', '\r\n some long text\r\n
consisting of two lines\r\n ']
['.default', '=', '\r\n even longer text\r\n
that can span even more\r\n than two lines\r\n
']
[['.summary', '=', '\r\n some long text\r\n
consisting of two lines\r\n '], ['.default', '=
', '\r\n even longer text\r\n that
can span even more\r\n than two lines\r\n
']]
i expected just two lists in the output, where is the third list of lists come from? is this by design? if i set listAllMatches = False in the template definition below
template = (Combine('.' + slug) + '=' + QuotedString('{', endQuoteChar = '}', multiline = True)).setResultsName('template')
the program outputs just templates (not lists/arrays) and there are just two templates instead of three. this is what i need. can someone explain why this is happening? i am trying to understand how pyparsing works.
thanks
konstantin
i was wrong.
template = (Combine('.' + slug) + '=' + QuotedString('{', endQuoteChar = '}', multiline = True)).setResultsName('template')
outputs all tokens in both templates as a flat list, i need to return the tokens that constitute each template just like the original code does (each token set in its own list or tuple), i only do not need the third array...
Hi,
I have been doing scripting with Lex/Yacc and regular expressions and I just run into pyparsing by accident yesterday. So, what do I need to try it out? I checked the download/installation pages, but I still wonder if I already need to have python installed? If so where to get one for Linux?
Thanks for helping me out.
You wonder correctly. You have to have Python installed, pyparsing is a class library written in Python.
Python is very often already installed in Linux, or if not, easily added using your Linux distributions package manager. I would suggest that you install Python 2.6, as I have done only minimal testing with Python 3.x.
-- Paul
I am new to parsing and have just come accross this tool which looks great.
For my work, I need to parse multiple lines of math equations with variables and user defined functions inside. For example, with 3 lines of string:
x = 1
y = Max(1,2)
x += y
I want to be able to get x = 3 and y = 2 at the end.
Below is my attempt on this. I am just trying to make a slight modification to the fourFn.py and SimpleCalc.py examples in the wiki page.
<span class="kw1">import</span> <span class="kw3">operator</span>
<span class="kw1">from</span> pyparsing <span class="kw1">import</span> Literal<span class="sy0">,</span>Word<span class="sy0">,</span>Combine<span class="sy0">,</span>Optional<span class="sy0">,</span>ZeroOrMore<span class="sy0">,</span>Forward<span class="sy0">,</span>nums<span class="sy0">,</span>\
alphas
<span class="kw1">from</span> <span class="kw3">collections</span> <span class="kw1">import</span> defaultdict
<span class="kw1">def</span> main<span class="br0">(</span><span class="br0">)</span>:
<span class="kw1">global</span> exprStack
exprStack <span class="sy0">=</span> <span class="br0">[</span><span class="br0">]</span>
<span class="kw1">global</span> varStack
varStack <span class="sy0">=</span> <span class="br0">[</span><span class="br0">]</span>
<span class="kw1">global</span> variables
variables <span class="sy0">=</span> defaultdict<span class="br0">(</span><span class="kw2">float</span><span class="br0">)</span>
l <span class="sy0">=</span> <span class="br0">[</span><span class="st0">'x=1'</span><span class="sy0">,</span><span class="st0">'y=abs(2)'</span><span class="sy0">,</span> <span class="st0">'x+=y'</span><span class="br0">]</span>
<span class="kw1">for</span> item <span class="kw1">in</span> l:
source <span class="sy0">=</span> item
L <span class="sy0">=</span> BNF<span class="br0">(</span><span class="br0">)</span>.<span class="me1">parseString</span><span class="br0">(</span> source <span class="br0">)</span>
<span class="kw1">print</span><span class="br0">(</span>L<span class="br0">)</span>
result <span class="sy0">=</span> evaluateStack<span class="br0">(</span> exprStack<span class="br0">[</span>:<span class="br0">]</span> <span class="br0">)</span>
<span class="kw1">print</span><span class="br0">(</span>varStack<span class="br0">)</span>
<span class="kw1">print</span><span class="br0">(</span>result<span class="br0">)</span>
variables<span class="br0">[</span>varStack.<span class="me1">pop</span><span class="br0">(</span><span class="br0">)</span><span class="br0">]</span><span class="sy0">=</span>result
<span class="kw1">print</span><span class="br0">(</span>variables<span class="br0">)</span>
<span class="kw1">def</span> Scalar<span class="br0">(</span>x<span class="br0">)</span>:
<span class="kw1">return</span> x
<span class="kw1">def</span> pushFirst<span class="br0">(</span> strg<span class="sy0">,</span> loc<span class="sy0">,</span> toks <span class="br0">)</span>:
exprStack.<span class="me1">append</span><span class="br0">(</span> toks<span class="br0">[</span><span class="nu0">0</span><span class="br0">]</span> <span class="br0">)</span>
<span class="kw1">def</span> assignVar<span class="br0">(</span> strg<span class="sy0">,</span> loc<span class="sy0">,</span> toks <span class="br0">)</span>:
varStack.<span class="me1">append</span><span class="br0">(</span> toks<span class="br0">[</span><span class="nu0">0</span><span class="br0">]</span> <span class="br0">)</span>
bnf <span class="sy0">=</span> <span class="kw2">None</span>
<span class="kw1">def</span> BNF<span class="br0">(</span><span class="br0">)</span>:
<span class="st0">'''
expop :: '^'
multop :: '*' | '/'
addop :: '+' | '-'
integer :: ['+' | '-'] '0'..'9'+
atom :: real | fn '(' expr ')' | '(' expr ')'
factor :: atom [ expop factor ]*
term :: factor [ multop factor ]*
expr :: term [ addop term ]*
iaddop :: '+='
assign :: '='
'''</span>
<span class="kw1">global</span> bnf
<span class="kw1">if</span> <span class="kw1">not</span> bnf:
point <span class="sy0">=</span> Literal<span class="br0">(</span> <span class="st0">'.'</span> <span class="br0">)</span>
fnumber <span class="sy0">=</span> Combine<span class="br0">(</span> Word<span class="br0">(</span> <span class="st0">'+-'</span>+nums<span class="sy0">,</span> nums <span class="br0">)</span> +
Optional<span class="br0">(</span> point + Optional<span class="br0">(</span> Word<span class="br0">(</span> nums <span class="br0">)</span> <span class="br0">)</span> <span class="br0">)</span><span class="br0">)</span>
ident <span class="sy0">=</span> Word<span class="br0">(</span>alphas+<span class="st0">'_'</span><span class="sy0">,</span> alphas+nums+<span class="st0">'_'</span><span class="br0">)</span>
plus <span class="sy0">=</span> Literal<span class="br0">(</span> <span class="st0">'+'</span> <span class="br0">)</span>
minus <span class="sy0">=</span> Literal<span class="br0">(</span> <span class="st0">'-'</span> <span class="br0">)</span>
mult <span class="sy0">=</span> Literal<span class="br0">(</span> <span class="st0">'*'</span> <span class="br0">)</span>
div <span class="sy0">=</span> Literal<span class="br0">(</span> <span class="st0">'/'</span> <span class="br0">)</span>
lpar <span class="sy0">=</span> Literal<span class="br0">(</span> <span class="st0">'('</span> <span class="br0">)</span>.<span class="me1">suppress</span><span class="br0">(</span><span class="br0">)</span>
rpar <span class="sy0">=</span> Literal<span class="br0">(</span> <span class="st0">')'</span> <span class="br0">)</span>.<span class="me1">suppress</span><span class="br0">(</span><span class="br0">)</span>
addop <span class="sy0">=</span> plus | minus
multop <span class="sy0">=</span> mult | div
expop <span class="sy0">=</span> Literal<span class="br0">(</span> <span class="st0">'^'</span> <span class="br0">)</span>
assign <span class="sy0">=</span> Literal<span class="br0">(</span> <span class="st0">'='</span> <span class="br0">)</span>
iaddop <span class="sy0">=</span> Literal<span class="br0">(</span><span class="st0">'+='</span><span class="br0">)</span>
expr <span class="sy0">=</span> Forward<span class="br0">(</span><span class="br0">)</span>
atom <span class="sy0">=</span> <span class="br0">(</span> fnumber | ident + lpar + expr + rpar | ident <span class="br0">)</span>.<span class="me1">setParseAction</span><span class="br0">(</span>pushFirst<span class="br0">)</span> | <span class="br0">(</span> lpar + expr.<span class="me1">suppress</span><span class="br0">(</span><span class="br0">)</span> + rpar <span class="br0">)</span>
<span class="co1"># by defining exponentiation as 'atom [ ^ factor ]...' instead of 'atom [ ^ atom ]...', we get right-to-left exponents, instead of left-to-righ</span>
<span class="co1"># that is, 2^3^2 = 2^(3^2), not (2^3)^2.</span>
factor <span class="sy0">=</span> Forward<span class="br0">(</span><span class="br0">)</span>
factor <span class="sy0">\<\<</span> atom + ZeroOrMore<span class="br0">(</span> <span class="br0">(</span> expop + factor <span class="br0">)</span>.<span class="me1">setParseAction</span><span class="br0">(</span> pushFirst <span class="br0">)</span> <span class="br0">)</span>
term <span class="sy0">=</span> factor + ZeroOrMore<span class="br0">(</span> <span class="br0">(</span> multop + factor <span class="br0">)</span>.<span class="me1">setParseAction</span><span class="br0">(</span> pushFirst <span class="br0">)</span> <span class="br0">)</span>
expr <span class="sy0">\<\<</span> term + ZeroOrMore<span class="br0">(</span> <span class="br0">(</span> addop + term <span class="br0">)</span>.<span class="me1">setParseAction</span><span class="br0">(</span> pushFirst <span class="br0">)</span> <span class="br0">)</span>
bnf <span class="sy0">=</span> ident.<span class="me1">setParseAction</span><span class="br0">(</span>assignVar<span class="br0">)</span> + <span class="br0">(</span>assign | iaddop.<span class="me1">setParseAction</span><span class="br0">(</span> pushFirst <span class="br0">)</span><span class="br0">)</span> + expr
<span class="kw1">return</span> bnf
<span class="co1"># map operator symbols to corresponding arithmetic operations</span>
opn <span class="sy0">=</span> <span class="br0">{</span> <span class="st0">'+'</span> : <span class="kw3">operator</span>.<span class="me1">add</span><span class="sy0">,</span>
<span class="st0">'-'</span> : <span class="kw3">operator</span>.<span class="me1">sub</span><span class="sy0">,</span>
<span class="st0">'*'</span> : <span class="kw3">operator</span>.<span class="me1">mul</span><span class="sy0">,</span>
<span class="st0">'/'</span> : <span class="kw3">operator</span>.<span class="me1">truediv</span><span class="sy0">,</span>
<span class="st0">'^'</span> : <span class="kw3">operator</span>.<span class="kw2">pow</span><span class="sy0">,</span>
<span class="st0">'+='</span> : <span class="kw3">operator</span>.<span class="me1">iadd</span><span class="br0">}</span>
fn <span class="sy0">=</span> <span class="br0">{</span> <span class="st0">'Max'</span> : <span class="kw2">max</span><span class="sy0">,</span>
<span class="st0">'Min'</span> : <span class="kw2">min</span><span class="sy0">,</span>
<span class="st0">'abs'</span> : <span class="kw2">abs</span><span class="sy0">,</span>
<span class="st0">'Scalar'</span> : Scalar<span class="br0">}</span>
<span class="kw1">def</span> evaluateStack<span class="br0">(</span> s <span class="br0">)</span>:
op <span class="sy0">=</span> s.<span class="me1">pop</span><span class="br0">(</span><span class="br0">)</span>
<span class="kw1">if</span> op <span class="kw1">in</span> <span class="st0">'+-*/^'</span> <span class="kw1">or</span> op <span class="kw1">in</span> <span class="st0">'+='</span>:
op2 <span class="sy0">=</span> evaluateStack<span class="br0">(</span> s <span class="br0">)</span>
op1 <span class="sy0">=</span> evaluateStack<span class="br0">(</span> s <span class="br0">)</span>
<span class="kw1">return</span> opn<span class="br0">[</span>op<span class="br0">]</span><span class="br0">(</span> op1<span class="sy0">,</span> op2 <span class="br0">)</span>
<span class="kw1">elif</span> op <span class="kw1">in</span> fn:
<span class="kw1">return</span> fn<span class="br0">[</span>op<span class="br0">]</span><span class="br0">(</span> evaluateStack<span class="br0">(</span> s <span class="br0">)</span> <span class="br0">)</span>
<span class="kw1">elif</span> op <span class="kw1">in</span> variables:
<span class="kw1">return</span> variables<span class="br0">[</span>op<span class="br0">]</span>
<span class="kw1">else</span>:
<span class="kw1">return</span> <span class="kw2">float</span><span class="br0">(</span> op <span class="br0">)</span>
<span class="kw1">if</span> __name__ <span class="sy0">==</span> <span class="st0">'__main__'</span>:
main<span class="br0">(</span><span class="br0">)</span>
But I have faced two issues.
-
The += operator is not working as expected. With input string as ['x=1','y=2', 'x+=y'], I still get x= 1 in the end. <-- defaultdict(<class 'float'>, {'x': 1.0, 'y': 2.0})
-
My user defined funciton name will have the same name convention as the variable name convention. The only difference is that the function will have parentheses followed. I tried to modify the atom definition but it is not working as expected. With input string as ['x=1','y=abs(2)', 'x+=y'], I got a ValueError: could not convert string to float: 'y'.
Would you help to shred some lights on how to fix the problem? Thanks.
For my work, there is no exponential operation, and this makes things easier. But on the other hand, the user defined function is exactly like a python function and can take argument such as list or keyword argument (name = value). i plan to directly write those functions in python, so i just need to call those functions with parsed argument list.
I have read more examples such as arith.py and excelExpr.py and discussion in this forum. And it seems infixNotation helps to simplify the definition of operators. But i am puzzled whether i should treat the brackets as an operator and if so how to do that.
I have tried a bit with new style of code like below by making list a kind of parseelement instead of an operator
<span class="kw1">from</span> pyparsing <span class="kw1">import</span> *
compOp <span class="sy0">=</span> oneOf<span class="br0">(</span><span class="st0">'\< \> \>= \<='</span><span class="br0">)</span>
multOp <span class="sy0">=</span> oneOf<span class="br0">(</span><span class="st0">'* /'</span><span class="br0">)</span>
addOp <span class="sy0">=</span> oneOf<span class="br0">(</span><span class="st0">'+ -'</span><span class="br0">)</span>
augassignOp <span class="sy0">=</span> Literal<span class="br0">(</span><span class="st0">'+='</span><span class="br0">)</span>
EQ<span class="sy0">,</span>LPAR<span class="sy0">,</span>RPAR<span class="sy0">,</span>LBRA<span class="sy0">,</span>RBRA <span class="sy0">=</span> <span class="kw2">map</span><span class="br0">(</span>Suppress<span class="sy0">,</span> <span class="st0">'=()[]'</span><span class="br0">)</span>
varname <span class="sy0">=</span> Regex<span class="br0">(</span>r<span class="st0">'[a-zA-Z_][a-zA-Z0-9_]*'</span><span class="br0">)</span>
fname <span class="sy0">=</span> Regex<span class="br0">(</span>r<span class="st0">'[a-zA-Z][a-zA-Z0-9_]*'</span><span class="br0">)</span>
decimalnumber <span class="sy0">=</span> Regex<span class="br0">(</span>r<span class="st0">'[+-]?<span class="es0">\d</span>+(<span class="es0">\.</span><span class="es0">\d</span>*)?'</span><span class="br0">)</span>
expr <span class="sy0">=</span> Forward<span class="br0">(</span><span class="br0">)</span>
listmaker <span class="sy0">=</span> Group<span class="br0">(</span>LBRA + delimitedList<span class="br0">(</span>expr<span class="br0">)</span> + RBRA<span class="br0">)</span><span class="br0">(</span><span class="st0">'list'</span><span class="br0">)</span>
funccall <span class="sy0">=</span> Group<span class="br0">(</span>fname<span class="br0">(</span><span class="st0">'fname'</span><span class="br0">)</span> + LPAR + Group<span class="br0">(</span>delimitedList<span class="br0">(</span>expr<span class="br0">)</span><span class="br0">)</span><span class="br0">(</span><span class="st0">'arglist'</span><span class="br0">)</span> + RPAR<span class="br0">)</span>
operand <span class="sy0">=</span> decimalnumber | listmaker | funccall | varname
expr <span class="sy0">\<\<=</span> infixNotation<span class="br0">(</span>operand<span class="sy0">,</span>
<span class="br0">[</span>
<span class="br0">(</span>multOp<span class="sy0">,</span> <span class="nu0">2</span><span class="sy0">,</span> opAssoc.<span class="me1">LEFT</span><span class="br0">)</span><span class="sy0">,</span>
<span class="br0">(</span>addOp<span class="sy0">,</span> <span class="nu0">2</span><span class="sy0">,</span> opAssoc.<span class="me1">LEFT</span><span class="br0">)</span><span class="sy0">,</span>
<span class="br0">(</span>compOp<span class="sy0">,</span> <span class="nu0">2</span><span class="sy0">,</span> opAssoc.<span class="me1">LEFT</span><span class="br0">)</span><span class="sy0">,</span>
<span class="co1"># (augassignOp, 2, opAssoc.LEFT),</span>
<span class="br0">]</span><span class="br0">)</span>
l <span class="sy0">=</span> <span class="br0">[</span><span class="st0">'cond(1,select(4,[2, 3]))'</span><span class="br0">]</span>
<span class="kw1">for</span> item <span class="kw1">in</span> l:
source <span class="sy0">=</span> item
expr.<span class="me1">runTests</span><span class="br0">(</span>source<span class="br0">)</span>
The parsing seems to work as I got below result:
cond<span class="br0">(</span><span class="nu0">1</span><span class="sy0">,</span><span class="kw3">select</span><span class="br0">(</span><span class="nu0">4</span><span class="sy0">,</span><span class="br0">[</span><span class="nu0">2</span><span class="sy0">,</span> <span class="nu0">3</span><span class="br0">]</span><span class="br0">)</span><span class="br0">)</span>
<span class="br0">[</span><span class="br0">[</span><span class="st0">'cond'</span><span class="sy0">,</span> <span class="br0">[</span><span class="st0">'1'</span><span class="sy0">,</span> <span class="br0">[</span><span class="st0">'select'</span><span class="sy0">,</span> <span class="br0">[</span><span class="st0">'4'</span><span class="sy0">,</span> <span class="br0">[</span><span class="st0">'2'</span><span class="sy0">,</span> <span class="st0">'3'</span><span class="br0">]</span><span class="br0">]</span><span class="br0">]</span><span class="br0">]</span><span class="br0">]</span><span class="br0">]</span>
<span class="br0">[</span><span class="nu0">0</span><span class="br0">]</span>:
<span class="br0">[</span><span class="st0">'cond'</span><span class="sy0">,</span> <span class="br0">[</span><span class="st0">'1'</span><span class="sy0">,</span> <span class="br0">[</span><span class="st0">'select'</span><span class="sy0">,</span> <span class="br0">[</span><span class="st0">'4'</span><span class="sy0">,</span> <span class="br0">[</span><span class="st0">'2'</span><span class="sy0">,</span> <span class="st0">'3'</span><span class="br0">]</span><span class="br0">]</span><span class="br0">]</span><span class="br0">]</span><span class="br0">]</span>
- arglist: <span class="br0">[</span><span class="st0">'1'</span><span class="sy0">,</span> <span class="br0">[</span><span class="st0">'select'</span><span class="sy0">,</span> <span class="br0">[</span><span class="st0">'4'</span><span class="sy0">,</span> <span class="br0">[</span><span class="st0">'2'</span><span class="sy0">,</span> <span class="st0">'3'</span><span class="br0">]</span><span class="br0">]</span><span class="br0">]</span><span class="br0">]</span>
<span class="br0">[</span><span class="nu0">0</span><span class="br0">]</span>:
<span class="nu0">1</span>
<span class="br0">[</span><span class="nu0">1</span><span class="br0">]</span>:
<span class="br0">[</span><span class="st0">'select'</span><span class="sy0">,</span> <span class="br0">[</span><span class="st0">'4'</span><span class="sy0">,</span> <span class="br0">[</span><span class="st0">'2'</span><span class="sy0">,</span> <span class="st0">'3'</span><span class="br0">]</span><span class="br0">]</span><span class="br0">]</span>
- arglist: <span class="br0">[</span><span class="st0">'4'</span><span class="sy0">,</span> <span class="br0">[</span><span class="st0">'2'</span><span class="sy0">,</span> <span class="st0">'3'</span><span class="br0">]</span><span class="br0">]</span>
- <span class="kw2">list</span>: <span class="br0">[</span><span class="st0">'2'</span><span class="sy0">,</span> <span class="st0">'3'</span><span class="br0">]</span>
- fname: <span class="st0">'select'</span>
- fname: <span class="st0">'cond'</span>
But i am puzzled how to pass the argument back to the function as a list. Also i have not figured out how to handle the keyword argument.
Also, i am still stuck at the augassignOp because i cannot use a lambda expression like l'ambda a, b: a = a + b' as shown in the airth.py example.
Would appreciate any feedback. Thanks.
The parsed data returned by pyparsing implements all the methods of a list so that it can be treated as a list (len, for i in result, result[0], etc.). If you must have a list (for JSON serialization, for example), you can use result.asList() and the parsed data will be converted to a nested list. Note that this will strip any defined results names, like' fname' or 'argslist'.
augAssignOp won't really fit into into the infixNotation model because '+=' isn't really an operator. You can't write 'x * (a += 14)'. The '+=' can only occur as part of an enhanced assignment statement. So you should really implement '+=' as another form of assignment:
EQ = Literal('=')
PLUS_EQ = Literal('+=')
assignmentExpr = varname('lhs') + EQ('operator') + arithExpr('rhs')
augmentedAssignmentExpr = varname('lhs') + PLUS_EQ('operator') + arithExpr('rhs')
and then have your overall parser match for 'assignment | augmentedAssignment'.
statement_parser = assignmentExpr | augmentedAssignemtExpr
You could attach a parse action to statement_parser that would look at the variable name in 'lhs', the value in 'rhs', and the assignment or augmented assignment based on whether 'operator' was '=' or '+='.
Thanks for the clarification. Would you mind to give a simple example on how to assign the token value on rhs to variable on lhs? I tried something like this:
assignmentExpr = Group(varname('lhs') + EQ('operator') + expr('rhs'))('assignment').addParseAction(lambda t: t[0] = t[-1])
but i got an error
SyntaxError: lambda cannot contain assignment
Also, what is the difference between addParseAction and setParseAction?
ah, i did not notice the tokens become a list if i use Group function. After removing the Group and use code like this, i got
def assignVar(t):
global vars_
vars_=defaultdict(float)
print('t0 is {}'.format(t[0]))
vars_[t[0]]=t[-1]
The var on lhs is updated with rhs value. If there is any better way, please let me know.
Another question is on the funccall. I would need to map many user function to python function. In the excelExpr.py example the funcCall is actually an aggregation using MatchFirst of various functions. This seems a bit tedious as we need to a new line each time we have a new function and then append the new function name in the funcCall line. Is there easier way such as just using a dictionary?
Sorry for all these many questions but i am glad to be able to make some progress little by little.
Will you be putting 1.5.2 up on the pypi anytime soon? I use easy_install and would really like to be able to fetch my versions from there.
Thanks!
jw
I was just packaging up 1.5.2 yesterday to upload, I should have it done sometime this weekend (including pyparsing_py3 module for Python 3 users, and support for IronPython2.0.1).
-- Paul
I see print statements in the code examples, why not add the info that one has to run <code>2to3 -w</code> before using Pyparsing on py 3.
Well, strictly speaking, pyparsing itself is Py2.6 thru Py 3.x compatible, so 2to3 isn't necessary to use pyparsing. The examples are somewhat lagging, I agree, but I'm not sure that is enough of an issue that one would say they have to run 2to3 before they could use pyparsing.
I'm open to posting a notice somewhere regarding the version dependency of the examples, is the wiki page itself sufficient? Many people install pyparsing using pip or easy_install and (sadly) never even see the examples or the docs, so I'm not sure there is anyplace in the install itself to make this point. But I can definitely add a header on the wiki page, and in the README file that co-resides with the examples.
Hi again,
Yes i see now that sourceforge has also py 3.x installers, thats great. I just installed it and it works :-)
I suggest to append: ' (including Python 3)' to 'supports Python versions 2.6 and later' above. That is more explicit.
The reason is, i have run into websites where 'supports 2.x' was said, but it was written before python 3 existed. Other website authors saying this did ignore python 3 because they didnt like it (seems there exist a few out there). These authors did not care that people will probably install their software on a python 3 box and then get errors.
Therefore i like when a website says explicitely: 'yes, it is python 3'. It removes these ambiguities.
I think the above description is clearer, thanks for the suggestion!
Paul
Will pyparsing work under 64-bit Windows Vista using 64-bit Python 2.5x?
Dinesh
Pyparsing is pure Python, and makes no assumptions on integer size (that I can think of). So there should be no problem in running pyparsing on 64-bit platforms of any kind.
I personally have run pyparsing on Windows, StrongARM embedded processor (running Linux), and Debian. Pyparsing is bundled with a number of Linux distributions. I've not gotten any reports that pyparsing doesn't run on 64-bit platforms.
-- Paul
Thanks Paul. So far, pyparsing is working under 64-bit Windows. Cheers. Dinesh
Try following:
- from pyparsing import *
- da = oneOf( 'aa' ).setName( 'Double A' )
- da.parseString( 'a' )
pyparsing.ParseException: Expected Double A (at char 0), (line:1, col:1
This is correct.
Let's see another case:
- from pyparsing import *
- da = oneOf( 'aa', caseless=True ).setName( 'Double A' )
- da.parseString( 'a' )
pyparsing.ParseException: Expected 'aa' (at char 0), (line:1, col:1)
Now the error msg is rather useless. It should report with the symbol name 'Double A'.
I think there could be many similar problems in error reporting. But this is the easiest way to reproduce it. In short, all error msg should be generated lazily.
def __init__( self ) :
self.msg = 'Expected ' + self.name # this is problem some
# We should have used
def msg( self ) :
return 'Expected ' + self.name
Shinewu -
Thanks for reporting this. This is an interesting little problem, actually. This circumstance comes out of an effort at optimizing run-time (that is, parse time for pyparsing) behavior, by pre-calculating as much as possible during parser construction, and just reusing the pre-calculated results at parse time. This is why pyparsing pre-constructs exception objects, and raises them again and again, rather than create new exceptions every time one is needed, just to have it GC'ed milliseconds later. Since pyparsing uses exceptions and the call stack to keep track of where it is in the grammar, and which alternatives have matched or not, pyparsing can generate many thousands of exceptions in a given application. By using a pre-constructed exception, we take advantage of the fact that, for any particular expression, the only part that changes all that often is the exception location - the error message of the exception doesn't really change at all at parse time, so it is a saving to pre-calculate this message before parsing begins.
In this particular example, we aren't quite seeing the lazy/non-lazy behavior you describe. Instead, it is the different implementation of oneOf, depending on whether it is caseless or not. When caseless is False, oneOf constructs a Regex of the alternatives, after reordering to prevent a short alternative from masking a longer one. When caseless is True, oneOf skips the Regex step, and creates a MatchFirst instead. MatchFirst has different exception raising behavior, in that it reraises the exception of the subexpression that matched the furthest. In your example, there is only one subexpcession, a CaselessLiteral('aa'), so that is the one that gets raised. The error message for this is just 'Expected 'aa'', so that is what we get.
I have a couple of ideas on how to address this, such as a lazy init of the error message, but I will also have to change the oneOf implementation in the caseless case, to generate a Regex instead of a MatchFirst. Otherwise, I'll just be back where I started.
Thanks for the feedback, I'll clean this up for the next release,
-- Paul
Thanks for the clarification. I assume you have some profiling data to support your design decision. The issue I discovered here has nothing to do lazy initialization of error msgs.
The ParseException object rarely gets generated more than a few dozen times. Most parsing programs give up after a few errors. Are you internally using these exceptions to signal something? If so, probably a redesign will help much more than caching the objects.
Just my two cents. Thanks a lot for the wonderful job.
Was there a resolution to this? I am implementing this method as well and receiving the same error.
This was fixed in 1.4.6. Here is the actual code that shinewu tried to post:
from pyparsing import *
da = oneOf( 'aa' ).setName( 'Double A' )
da.parseString( 'a' )
print da,type(da)
With version 1.4.6 or higher, prints this exception:
pyparsing.ParseException: Expected Double A (at char 0), (line:1, col:1)
As you can see, the exception message is giving the right name for the missing expression.
-- Paul
So, as newbie to Pyparsing, how do I handel this exception type so that the program will continue to parse is the output is infact grabbing what I am looking at?
Sorry for the ambigious post. Got bothered while posting. I am using the following code:
from pyparsing import *
import os
import sys
import string
def subr()
da = oneOf( ('aa bb cc'), caseless=True ).setResultsName( 'Double Leters' )
db = oneOf( ('11 22 33'), caseless=True ).setResultsName( 'Double Numbs' )
all = OneOrMore(da | db)
return all
somefile = open('somefile.txt', 'r')
for line in somefile:
results = subr().parseString(line)
print results
And then I receive the above noted exception.
What does somefile.txt contain? Can you post it as well, or inline it in your sample code? By using parseString, pyparsing assumes that somefile.txt contains nothing else but double letters and double numbers. If you are trying to extract the doubles from a file containing other stuff, try using scanString or searchString (searchString is simpler to use, it is a simple wrapper around the generator function scanString).
One other note, you are calling setResultsName in your example, when the previous examples were calling setName. There is a big difference in these two methods.
-- Paul
#
# RC-6 config file
#
# source: http://home.hccnet.nl/m.majoor/projects__remote_control.htm
# http://home.hccnet.nl/m.majoor/pronto.pdf
#
# used by: Philips
#
#########
#
# Philips Media Center Edition remote control
# For use with the USB MCE ir receiver
#
# Dan Conti dconti|acm.wwu.edu
#
# Updated with codes for MCE 2005 Remote additional buttons
# *, #, Teletext, Red, Green, Yellow & Blue Buttons
# Note: TV power button transmits no code until programmed.
# Updated 12th September 2005
# Graham Auld - mce|graham.auld.me.uk
#
# Radio, Print, RecTV are only available on the HP Media Center remote control
#
begin remote
name mceusb
bits 16
flags RC6|CONST_LENGTH
eps 30
aeps 100
header 2667 889
one 444 444
zero 444 444
pre_data_bits 21
pre_data 0x37FF0
gap 105000
toggle_bit 22
rc6_mask 0x100000000
begin codes
Blue 0x00007ba1
Yellow 0x00007ba2
Green 0x00007ba3
Red 0x00007ba4
Teletext 0x00007ba5
# starts at af
Radio 0x00007baf
Print 0x00007bb1
Videos 0x00007bb5
Pictures 0x00007bb6
RecTV 0x00007bb7
Music 0x00007bb8
TV 0x00007bb9
# no ba - d8
Guide 0x00007bd9
LiveTV 0x00007bda
DVD 0x00007bdb
Back 0x00007bdc
OK 0x00007bdd
Right 0x00007bde
Left 0x00007bdf
Down 0x00007be0
Up 0x00007be1
Star 0x00007be2
Hash 0x00007be3
Replay 0x00007be4
Skip 0x00007be5
Stop 0x00007be6
Pause 0x00007be7
Record 0x00007be8
Play 0x00007be9
Rewind 0x00007bea
Forward 0x00007beb
ChanDown 0x00007bec
ChanUp 0x00007bed
VolDown 0x00007bee
VolUp 0x00007bef
More 0x00007bf0
Mute 0x00007bf1
Home 0x00007bf2
Power 0x00007bf3
Enter 0x00007bf4
Clear 0x00007bf5
Nine 0x00007bf6
Eight 0x00007bf7
Seven 0x00007bf8
Six 0x00007bf9
Five 0x00007bfa
Four 0x00007bfb
Three 0x00007bfc
Two 0x00007bfd
One 0x00007bfe
Zero 0x00007bff
end codes
end remote
That is the something.txt
The goal is to find the key code names as you can probably tell and store them in a reportable manner to be handeled later in another app I am writing.
Thanks!
Also any code suggestions you make I will be glad to use/test/hear as I am just a humble newb to this.
Thanks!
Ooooh, this is a juicy-looking format for a full parser, but I am sorely time-constrained just now. However, here is a short program to scan such a file and extract all entries of the form '<word> <hex-constant>' (actually, this much you could do with re's, but if you use pyparsing now, it can expand to parse the full file when you are ready). Also, this little example will skip over any entries in comments, which is a bit tricky using just re's.
-- Paul
from pyparsing import *
rc6config = file('somefile.txt').read()
# define expression for '\<word\> \<hex-constant\>'
hexConstant = Combine('0x' + Word(hexnums))
keydef = Word(alphas)('keyname') + hexConstant('value')
# define a comment, and set our keydef to ignore them
rc6comment = '#' + restOfLine
keydef.ignore( rc6comment )
# use searchString to skim through the file and return a list
# of matches, each is a separate ParseResults, with corresponding
# labels
for key in keydef.searchString( rc6config ):
print key.keyname, '=', key.value
Prints:
data = 0x37FF0
mask = 0x100000000
Blue = 0x00007ba1
Yellow = 0x00007ba2
Green = 0x00007ba3
Red = 0x00007ba4
Teletext = 0x00007ba5
Radio = 0x00007baf
Print = 0x00007bb1
Videos = 0x00007bb5
Pictures = 0x00007bb6
RecTV = 0x00007bb7
Music = 0x00007bb8
TV = 0x00007bb9
Guide = 0x00007bd9
LiveTV = 0x00007bda
DVD = 0x00007bdb
Back = 0x00007bdc
...
Oh, by the way, this example uses the new 1.4.7 syntax for defining results names. If you are using 1.4.6, change the keydef to:
keydef = Word(alphas).setResultsName('keyname') +
hexConstant.setResultsName('value')
-- Paul
Great, Thanks for the great Example. Got me where I needed to go!
Thanks!
hi, I think this is not as expected:
- from lib.pyparsing import *
- Stream=Forward()
- Stream\<\
- Stream.parseString('myc(114)r(11)dd')
that's ok, but:
- from lib.pyparsing import *
- Stream=Forward()
- Stream\<\
- Stream.parseString('myc(114)r(11)dd')
now what's the matter?
thx
Good catch - looks like Combine doesn't like recursion! Would you mind reporting this as a bug on SourceForge? Then I'll be sure to take care of it properly.
BTW, is there any reason you are using recursion instead of just using one of the iterative forms, such as OneOrMore? I notice that many folks porting BNF and EBNF grammars to pyparsing tend to use recursion, as in:
item :: blah blah
listOfItems :: item | item listOfItems
The pyparsing form of this is just:
listOfItems = OneOrMore(item)
(If items are separated by commas or some other non-whitespace delimiter, use delimitedList.)
For your code, these perform as expected:
print OneOrMore( Word(alphas) | '('+Word(nums)+')').parseString('myc(114)r(11)dd')
print Combine(OneOrMore( Word(alphas) | '('+Word(nums)+')')).parseString('myc(114)r(11)dd')
Giving:
['myc', '(', '114', ')', 'r', '(', '11', ')', 'dd']
['myc(114)r(11)dd']
I'll still look into that Combine problem, though!
-- Paul
Thanks.
I've looked into this a little further, and this turns out to be fairly tricky. The problem stems from pyparsing's whitespace-skipping behavior. Combine turns off whitespace skipping in embedded expressions, but first makes a copy of them so as not to change the subexpressions' default behavior. Unfortunately, at the time this is done, the embedded Forward has not yet had its contents assigned, so the copy contains no expression to be matched - Forwards with no content always fail, therefore, the Combine fails.
I'll see if I can make Forwards a little smarter when they get copied.
-- Paul
Yes, I think you could also enable using Forward with setResultsName().
I have this problem fixed, and it will be in the next pyparsing release (1.4.4). And it does address the setResultsName problem with Forwards also.
ETA on this release is probably mid-October or so.
-- Paul