Some tests fail with pypy #165

Closed
onyxfish opened this Issue May 2, 2012 · 14 comments

2 participants

@onyxfish

No description provided.

@msabramo msabramo added a commit to msabramo/csvkit that referenced this issue May 12, 2012
@msabramo msabramo Fix for the fact that some versions of the csv module (apparently
including some bundled with pypy) misspell the word "larger" as
"larget". This fixes one pypy test failure mentioned in GH-165.
1ad45af
@msabramo

I get one error with pypy now:

[last: 0] marca@SCML-MarcA:~/dev/git-repos/csvkit$ .tox/pypy/bin/python -m unittest tests.test_convert.test_dbf.TestDBF
E
======================================================================
ERROR: test_dbf (tests.test_convert.test_dbf.TestDBF)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_convert/test_dbf.py", line 10, in test_dbf
    output = dbase.dbf2csv(f)
  File "csvkit/convert/dbase.py", line 23, in dbf2csv
    for i, d in enumerate(row):
TypeError: 'function' object is not iterable

----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (errors=1)

but I suspect that it might be a bug in the dbf module, because it has some similar looking test failures:

======================================================================
ERROR: test_add_field_to_null (__main__.Test_Dbf_Records)
adding a normal field to a table with null fields
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_dbf.py", line 3124, in test_add_field_to_null
    table.add_fields('tenure N(3,0)')
  File "/Users/marca/src/dbf-0.93.000/dbf.py", line 3308, in add_fields
    self.append(record.scatter_fields())
AttributeError: 'function' object has no attribute 'scatter_fields'
@onyxfish

Ahhh, interesting. Yes, I've def been seeing this test failure, but didn't see an obvious fix. It's highly unlikely that the enumerate function is implemented differently in PyPy, so I suspect it is something that would need to be fixed within the dbf module.

@msabramo

I'm not sure why, but it seems to be passing now for me:

[last: 0] marca@scml-marca:~/dev/git-repos/csvkit$ PAGER=cat git log -n 1
commit 3323d4542fbcccf081ccf39f0f33bc533d96de91
Author: Marc Abramowitz <marc@marc-abramowitz.com>
Date:   Tue May 15 13:40:44 2012 -0700

    Add a setup.cfg file that tells nosetests to ignore dbf.py.
    This fixes GH-169 for me.
[last: 0] marca@scml-marca:~/dev/git-repos/csvkit$ tox
_______________________________________________________________________ [tox sdist] ________________________________________________________________________
[TOX] ***creating sdist package
[TOX] /Users/marca/dev/git-repos/csvkit$ /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python setup.py sdist --formats=zip --dist-dir .tox/dist >.tox/log/0.log
[TOX] ***copying new sdistfile to '/Users/marca/.tox/distshare/csvkit-0.4.5.zip'
____________________________________________________________________ [tox testenv:py26] ____________________________________________________________________
[TOX] ***reusing existing matching virtualenv py26
[TOX] ***upgrade-installing sdist
[TOX] /Users/marca/dev/git-repos/csvkit/.tox/py26/log$ ../bin/pip install --download-cache=/Users/marca/dev/git-repos/csvkit/.tox/_download /Users/marca/dev/git-repos/csvkit/.tox/dist/csvkit-0.4.5.zip -U --no-deps >6.log
[TOX] /Users/marca/dev/git-repos/csvkit$ .tox/py26/bin/nosetests
........................................................................................................................................................
----------------------------------------------------------------------
Ran 152 tests in 0.559s

OK
____________________________________________________________________ [tox testenv:py27] ____________________________________________________________________
[TOX] ***reusing existing matching virtualenv py27
[TOX] ***upgrade-installing sdist
[TOX] /Users/marca/dev/git-repos/csvkit/.tox/py27/log$ ../bin/pip install --download-cache=/Users/marca/dev/git-repos/csvkit/.tox/_download /Users/marca/dev/git-repos/csvkit/.tox/dist/csvkit-0.4.5.zip -U --no-deps >10.log
[TOX] /Users/marca/dev/git-repos/csvkit$ .tox/py27/bin/nosetests
........................................................................................................................................................
----------------------------------------------------------------------
Ran 152 tests in 0.516s

OK
____________________________________________________________________ [tox testenv:pypy] ____________________________________________________________________
[TOX] ***reusing existing matching virtualenv pypy
[TOX] ***upgrade-installing sdist
[TOX] /Users/marca/dev/git-repos/csvkit/.tox/pypy/log$ ../bin/pip install --download-cache=/Users/marca/dev/git-repos/csvkit/.tox/_download /Users/marca/dev/git-repos/csvkit/.tox/dist/csvkit-0.4.5.zip -U --no-deps >23.log
[TOX] /Users/marca/dev/git-repos/csvkit$ .tox/pypy/bin/nosetests
........................................................................................................................................................
----------------------------------------------------------------------
Ran 152 tests in 1.400s

OK
______________________________________________________________________ [tox summary] _______________________________________________________________________
[TOX] py26: commands succeeded
[TOX] py27: commands succeeded
[TOX] pypy: commands succeeded
[TOX] congratulations :)

[last: 0] marca@scml-marca:~/dev/git-repos/csvkit$ .tox/pypy/bin/pip freeze
SQLAlchemy==0.6.6
argparse==1.2.1
csvkit==0.4.5
dbf==0.93.000
distribute==0.6.24
nose==1.1.2
openpyxl==1.5.7
python-dateutil==1.5
wsgiref==0.1.2
xlrd==0.7.1

Perhaps one of the other fixes also fixed pypy support?

@onyxfish

Hmmm, I can't replicate locally. I updated my env to match your versions of dbf and nose, but no luck. Anything else you might have tweaked? Can you tell me the output of:

.tox/pypy/bin/python --version

@onyxfish

Hold that thought, I only installed the updated reqs locally, not in my tox environments. Will test again once they are rebuilt.

@onyxfish

This still fails with PyPy for me, even using dbf==0.93.000. Really curious what you might have changed. This isn't because of the pull request you submitted to ignore dbf.py, is it? (I don't want to just ignore the test.)

@onyxfish

Now that I look back I see you pasted your git log and that this does include the pull request you sent. I'm clear on why this is fixing it. Is nose simply ignoring any exception raised within the dbf module?

@msabramo

I could be wrong but my understanding of that nose --ignore-files option is that it makes nose not try to import dbf.py. Without that ,nose was importing dbf.py and evaluating the default locale before I had a chance to override it in os.environ. I can test later, but I think dbf.py later gets imported when a test needs it. I think the ignore-files stuff is just to prevent it from getting prematurely imported.

As for this problem, I am not sure but it might be similar - evaluating dbf.py before something else might trigger the problem.

@msabramo

I can reproduce the TypeError: 'function' object is not iterable exception by doing as follows:

~/dev/git-repos/csvkit$ mv setup.cfg _setup.cfg
~/dev/git-repos/csvkit$ LC_ALL="" .tox/pypy/bin/nosetests
.....................E...................................................................................................................................
======================================================================
ERROR: test_dbf (tests.test_convert.test_dbf.TestDBF)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/marca/dev/git-repos/csvkit/tests/test_convert/test_dbf.py", line 10, in test_dbf
    output = dbase.dbf2csv(f)
  File "/Users/marca/dev/git-repos/csvkit/csvkit/convert/dbase.py", line 23, in dbf2csv
    for i, d in enumerate(row):
TypeError: 'function' object is not iterable

----------------------------------------------------------------------
Ran 153 tests in 1.443s
@msabramo

Oh crap. I think you're right. I think adding that --ignore-files option is causing it to skip test_dbf:

~/dev/git-repos/csvkit$ LC_ALL="" .tox/pypy/bin/nosetests -s -I dbf.py -v -v -v 2>&1 | grep test_dbf
nose.selector: DEBUG: test_dbf.py matches ignoreFiles pattern; skipped
nose.selector: DEBUG: test_dbf.pyc matches ignoreFiles pattern; skipped
@msabramo

OK, the change in #171 is not good. I just closed it.

@onyxfish

Great, thanks for following up on that. I had a suspicion that wasn't quite what we wanted.

@msabramo

OK, #172 fixes the encoding-related error, but it does NOT fix this issue.

[last: 0] marca@SCML-MarcA:~/dev/git-repos/csvkit$ PAGER=cat git log -n 1
commit 4fd79bf80cdbeb6f621a263e92bf9077487b16ec
Author: Marc Abramowitz <marc@marc-abramowitz.com>
Date:   Tue May 15 17:16:19 2012 -0700

    This fixes an encoding-related test failure ("TypeError: expected string, got NoneType
    object") in test_dbf. It is a replacement for the bad approach in #171.
[last: 0] marca@SCML-MarcA:~/dev/git-repos/csvkit$ LC_ALL="C" .tox/pypy/bin/nosetests      
.....................E...................................................................................................................................
======================================================================
ERROR: test_dbf (tests.test_convert.test_dbf.TestDBF)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/marca/dev/git-repos/csvkit/tests/test_convert/test_dbf.py", line 13, in test_dbf
    output = dbase.dbf2csv(f)
  File "/Users/marca/dev/git-repos/csvkit/csvkit/convert/dbase.py", line 23, in dbf2csv
    for i, d in enumerate(row):
TypeError: 'function' object is not iterable

----------------------------------------------------------------------
Ran 153 tests in 1.201s

FAILED (errors=1)
@onyxfish

Marc, I've worked on this a bit and currently have no failing tests in any of my environments, so I'm going to close this out along with your pull request #172. Please feel free to reopen if you're still having issues.

@onyxfish onyxfish closed this Aug 14, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment