Unknown error generating cffi file

[joxeankoret]

Hi!

I'm trying to build PyVex in Ubuntu 14.04 (32bits) and I'm getting the following error:

$ python setup.py build
Proper 'develop' support unavailable.
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'install_requires'
  warnings.warn(msg)
running build
Building libVEX
EXTRA_CFLAGS="-fPIC -w" make -f Makefile-gcc
make[1]: Entering directory `/path/to/pyvex/vex-dev'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/path/to/pyvex/vex-dev'
Building pyvex-static
make: `pyvex_static.so' is up to date.
Creating CFFI defs file
No handlers could be found for logger "cffier"
Traceback (most recent call last):
  File "make_ffi.py", line 77, in <module>
    doit(sys.argv[1])
  File "make_ffi.py", line 67, in doit
    new_good = find_good_scan(good, remaining[1:])
  File "make_ffi.py", line 51, in find_good_scan
    return find_good_scan(known_good, questionable[:fail-2-len(known_good)])
  File "make_ffi.py", line 49, in find_good_scan
    raise Exception("Unknown error")
Exception: Unknown error
error: Unable to generate cffi file.

Am I doing something wrong? Thanks!

[rhelmot]

@zardus: at the very least, it should be Exception("Unknown error %s" % e.message) so that debugging isn't a nightmare 😛

@joxeankoret: what did you do involving the vex_dev path? That's not what it usually is, iirc, and if you've changed it so it'll use that folder, you might need to change a few more copies of that path, this time related to the cffi-generation procedure. We parse some of the header files from vex in order to build pyvex, and if it can't find vex in the usual spot it's gonna complain.

[joxeankoret]

I haven't done anything special. In another x86_64 box it builds perfectly while in that x86 VM it doesn't. I tried to remove the vex symlink so it downloads https://github.com/angr/vex/archive/dev.tar.gz

Am I doing something wrong? Do I need to specify some specific path somehow?

[joxeankoret]

Update: I modified the setup.py script so it shows what is happening:

$ LANG=C python setup.py build
running build
Building libVEX
EXTRA_CFLAGS="-fPIC -w" make -f Makefile-gcc
make[1]: Entering directory `/home/joxean/devel/pyvex/vex-dev'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/home/joxean/devel/pyvex/vex-dev'
Building pyvex-static
make: `pyvex_static.so' is up to date.
Creating CFFI defs file
No handlers could be found for logger "cffier"
Traceback (most recent call last):
  File "make_ffi.py", line 79, in <module>
    doit(sys.argv[1])
  File "make_ffi.py", line 64, in doit
    good = find_good_scan([], ffi_lines)
  File "make_ffi.py", line 32, in find_good_scan
    ffi.cdef('\n'.join(known_good + questionable))
  File "/usr/lib/python2.7/dist-packages/cffi/api.py", line 104, in cdef
    self._parser.parse(csource, override=override, packed=packed)
  File "/usr/lib/python2.7/dist-packages/cffi/cparser.py", line 157, in parse
    self._internal_parse(csource)
  File "/usr/lib/python2.7/dist-packages/cffi/cparser.py", line 163, in _internal_parse
    ast, macros = self._parse(csource)
  File "/usr/lib/python2.7/dist-packages/cffi/cparser.py", line 127, in _parse
    self.convert_pycparser_error(e, csource)
  File "/usr/lib/python2.7/dist-packages/cffi/cparser.py", line 149, in convert_pycparser_error
    raise api.CDefError(msg)
cffi.api.CDefError: cannot parse "extern int setjmp (jmp_buf __env) __attribute__ ((__nothrow__));"
:1147:35: before: __attribute__
error: Unable to generate cffi file.

[zardus]

Hi Joxean,

It's possible that you're bypassing the dependency resolution, and your
CFFI or pycparser version is too old. Can you manually update them (pip install -U cffi pycparser, for example) and see if the issue goes away?
On Jun 8, 2015 7:28 AM, "Joxean" notifications@github.com wrote:

Update: I modified the setup.py script so it shows what is happening:

$ LANG=C python setup.py build
running build
Building libVEX
EXTRA_CFLAGS="-fPIC -w" make -f Makefile-gcc
make[1]: Entering directory /home/joxean/devel/pyvex/vex-dev' make[1]: Nothing to be done forall'.
make[1]: Leaving directory /home/joxean/devel/pyvex/vex-dev' Building pyvex-static make:pyvex_static.so' is up to date.
Creating CFFI defs file
No handlers could be found for logger "cffier"
Traceback (most recent call last):
File "make_ffi.py", line 79, in
doit(sys.argv[1])
File "make_ffi.py", line 64, in doit
good = find_good_scan([], ffi_lines)
File "make_ffi.py", line 32, in find_good_scan
ffi.cdef('\n'.join(known_good + questionable))
File "/usr/lib/python2.7/dist-packages/cffi/api.py", line 104, in cdef
self._parser.parse(csource, override=override, packed=packed)
File "/usr/lib/python2.7/dist-packages/cffi/cparser.py", line 157, in parse
self._internal_parse(csource)
File "/usr/lib/python2.7/dist-packages/cffi/cparser.py", line 163, in _internal_parse
ast, macros = self._parse(csource)
File "/usr/lib/python2.7/dist-packages/cffi/cparser.py", line 127, in _parse
self.convert_pycparser_error(e, csource)
File "/usr/lib/python2.7/dist-packages/cffi/cparser.py", line 149, in convert_pycparser_error
raise api.CDefError(msg)
cffi.api.CDefError: cannot parse "extern int setjmp (jmp_buf __env) attribute ((nothrow));"
:1147:35: before: attribute
error: Unable to generate cffi file.

—
Reply to this email directly or view it on GitHub
#14 (comment).

[joxeankoret]

Unfortunately, it doesn't, the same happens.

[zardus]

Bummer... Could you post the output of pip freeze? I'll want to recreate your environment to see if I can get the issue to reoccur.

In the meantime, I looked closer into my code, and that exception should be caught. In the code, there is:

    try:
        ffi.cdef('\n'.join(known_good + questionable))
        return questionable
    except AssertionError:
        return [ ]
    except cffi.CDefError as e:
        if str(e).count(':') >= 2:
            fail = int(str(e).split('\n')[1].split(':')[1])
        elif 'unrecognized construct' in str(e):
            fail = int(str(e).split()[1][:-1])
        elif 'end of input' in str(e):
            return find_good_scan(known_good, questionable[:-1])
        else:
            raise Exception("Unknown error")

That first case should catch it (if str(e).count(':') >= 2) and parse that error, but for some reason, it doesn't. In fact, on my machine, it does catch it properly. I made the following change to make_ffi.py:

diff --git a/make_ffi.py b/make_ffi.py
index 3031f10..ba5f071 100644
--- a/make_ffi.py
+++ b/make_ffi.py
@@ -34,6 +34,7 @@ def find_good_scan(known_good, questionable):
     except AssertionError:
         return [ ]
     except cffi.CDefError as e:
+        print repr(str(e))
         if str(e).count(':') >= 2:
             fail = int(str(e).split('\n')[1].split(':')[1])
         elif 'unrecognized construct' in str(e):

I can see it print that error and properly continue. Could you apply that and see what it prints?

[joxeankoret]

Hi,

This is the list of pip installed packages:

adium-theme-ubuntu==0.3.4
apt-xapian-index==0.45
archinfo==0.2
capstone==4.0
cffi==0.8.2
chardet==2.0.1
colorama==0.2.5
command-not-found==0.3
Cython==0.20.1.post0
debtagshw==0.1
defer==1.0.6
dirspec==13.10
duplicity==0.6.23
html5lib==0.999
httplib2==0.8
lockfile==0.8
lxml==3.3.3
oauthlib==0.6.1
oneconf==0.3.7
PAM==0.4.2
pexpect==3.1
Pillow==2.3.0
piston-mini-client==0.7.5
ply==3.4
pycparser==2.13
pycrypto==2.6.1
pycups==1.9.66
pycurl==7.19.3
pygobject==3.12.0
pyOpenSSL==0.13
pyserial==2.6
pysmbc==1.0.14.1
python-apt===0.9.3.5ubuntu1
python-debian===0.1.21-nmu2ubuntu2
pyxdg==0.25
reportlab==3.0
requests==2.2.1
sessioninstaller==0.0.0
six==1.5.2
software-center-aptd-plugins==0.0.0
system-service==0.1.6
Twisted-Core==13.2.0
Twisted-Web==13.2.0
unity-lens-photos==1.0
urllib3==1.7.1
wheel==0.24.0
xdiagnose===3.6.3build2
zope.interface==4.0.5

And this is the ouput after applying your patch:

Building pyvex-static
make: «pyvex_static.so» está actualizado.
Creating CFFI defs file
No handlers could be found for logger "cffier"
'cannot parse "extern int setjmp (jmp_buf __env) __attribute__ ((__nothrow__));"\n:1147:35: before: __attribute__'
'line 33: unrecognized construct'
'cannot parse "return (Bool)r;"\n:33:4: before: return'
'cannot parse "extern int setjmp (jmp_buf __env) __attribute__ ((__nothrow__));"\n:1143:35: before: __attribute__'
'line 33: unrecognized construct'
'cannot parse "x &= 0xFF;"\n:33:6: before: &='
'cannot parse "return (UChar)x;"\n:33:4: before: return'
'cannot parse "extern int setjmp (jmp_buf __env) __attribute__ ((__nothrow__));"\n:1139:35: before: __attribute__'
'line 33: unrecognized construct'
'cannot parse "x &= 0xFF;"\n:33:6: before: &='
'cannot parse "return (HChar)x;"\n:33:4: before: return'
'cannot parse "extern int setjmp (jmp_buf __env) __attribute__ ((__nothrow__));"\n:1135:35: before: __attribute__'
'line 33: unrecognized construct'
'cannot parse "x &= 0xFFFF;"\n:33:6: before: &='
'cannot parse "return (UShort)x;"\n:33:4: before: return'
'cannot parse "extern int setjmp (jmp_buf __env) __attribute__ ((__nothrow__));"\n:1131:35: before: __attribute__'
'line 33: unrecognized construct'
'cannot parse "x &= 0xFFFF;"\n:33:6: before: &='
'cannot parse "return (Short)x;"\n:33:4: before: return'
'cannot parse "extern int setjmp (jmp_buf __env) __attribute__ ((__nothrow__));"\n:1127:35: before: __attribute__'
'line 33: unrecognized construct'
'cannot parse "x &= 0xFFFFFFFFLL;"\n:33:6: before: &='
'cannot parse "return (UInt)x;"\n:33:4: before: return'
'cannot parse "extern int setjmp (jmp_buf __env) __attribute__ ((__nothrow__));"\n:1123:35: before: __attribute__'
'line 781: unrecognized construct'
'cannot parse "return e->tag == Iex_VECRET || e->tag == Iex_BBPTR;"\n:781:4: before: return'
'cannot parse "extern int setjmp (jmp_buf __env) __attribute__ ((__nothrow__));"\n:1120:35: before: __attribute__'
'line 819: unrecognized construct'
'cannot parse "return toBool(e->tag == Iex_RdTmp || e->tag == Iex_Const);"\n:819:4: before: return'
'cannot parse "extern int setjmp (jmp_buf __env) __attribute__ ((__nothrow__));"\n:1117:35: before: __attribute__'
Traceback (most recent call last):
  File "make_ffi.py", line 79, in <module>
    doit(sys.argv[1])
  File "make_ffi.py", line 69, in doit
    new_good = find_good_scan(good, remaining[1:])
  File "make_ffi.py", line 53, in find_good_scan
    return find_good_scan(known_good, questionable[:fail-2-len(known_good)])
  File "make_ffi.py", line 50, in find_good_scan
    raise Exception("Unknown error %s" % e.message)
Exception: Unknown error unsupported expression: expected a simple numeric constant
error: Unable to generate cffi file.

[zardus]

Interesting... I don't see the expected a simple numeric constant error.

One thing that jumps out at me is your cffi version. 0.8.2 is the version shipped with ubuntu, and I am fairly sure that it is too old, at least for how we're generating the FFI definitions file. Interestingly enough, it looks like your pycparser is up-to-date; just cffi that's behind. Can you try upgrading it to the latest (1.0.3)? I think it should fix the issue.

Using pip install -e instead of python setup.py develop should do the dependency resolution for you, too.

[joxeankoret]

I'm getting a different error now :/

Traceback (most recent call last):
  File "make_ffi.py", line 79, in <module>
    doit(sys.argv[1])
  File "make_ffi.py", line 69, in doit
    new_good = find_good_scan(good, remaining[1:])
  File "make_ffi.py", line 53, in find_good_scan
    return find_good_scan(known_good, questionable[:fail-2-len(known_good)])
  File "make_ffi.py", line 32, in find_good_scan
    ffi.cdef('\n'.join(known_good + questionable))
  File "/usr/local/lib/python2.7/dist-packages/cffi/api.py", line 107, in cdef
    self._parser.parse(csource, override=override, packed=packed)
  File "/usr/local/lib/python2.7/dist-packages/cffi/cparser.py", line 167, in parse
    self._internal_parse(csource)
  File "/usr/local/lib/python2.7/dist-packages/cffi/cparser.py", line 201, in _internal_parse
    realtype = self._get_type(decl.type, name=decl.name)
  File "/usr/local/lib/python2.7/dist-packages/cffi/cparser.py", line 386, in _get_type
    return self._get_struct_union_enum_type('struct', type, name)
  File "/usr/local/lib/python2.7/dist-packages/cffi/cparser.py", line 543, in _get_struct_union_enum_type
    type = self._get_type(decl.type, partial_length_ok=True)
  File "/usr/local/lib/python2.7/dist-packages/cffi/cparser.py", line 338, in _get_type
    typenode.dim, partial_length_ok=partial_length_ok)
  File "/usr/local/lib/python2.7/dist-packages/cffi/cparser.py", line 607, in _parse_constant
    "simple numeric constant" % exprnode.coord.line)
cffi.api.FFIError: :1108: unsupported expression: expected a simple numeric constant
    *** Err:     unsigned long int __val[(1024 / (8 * sizeof (unsigned long int)))];
error: Unable to generate cffi file.

I will try to find myself the answer because it seems to be very-very specific to my case.

[joxeankoret]

Forget it, I had modified the make_ffi.py script as to raise always the errors so I could see. After upgrading cffi myself manually it works as expected. Thanks for all the help!

[zardus]

No problem; I'm glad it works!