Issue with reverse complement test in v0.5.1 on fresh install of python 2.7

[ian1roberts]

Dear Sir,

I've experienced a strange issue with reverse complementing sequences using the latest pyfasta from pypi.

I'm using python 2.7.1 and set up a fresh virtualenv to replicate the error I've been facing. A snippet of the test output is at the end.

I came to the issue following an accidental upgrade from pyfasta 0.4.5 (pip install pyfasta picks latest !)

I was able to trace my problem to reverse complementing sequences, but not sufficiently skilled to determine exactly what caused the issue in the code.

I've temporarily fixed my requirement by downgrading to 0.4.5 (pip install pyfasta==0.4.5) and tests run fine!

Many thanks, Ian

(reflexdev)[ir210@beast pyfasta]$ python setup.py nosetests
running nosetests
running egg_info
writing pyfasta.egg-info/PKG-INFO
writing top-level names to pyfasta.egg-info/top_level.txt
writing dependency_links to pyfasta.egg-info/dependency_links.txt
writing entry points to pyfasta.egg-info/entry_points.txt
reading manifest file 'pyfasta.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pyfasta.egg-info/SOURCES.txt'
running build_ext
nose.plugins.cover: ERROR: Coverage not available: unable to import coverage module
.F..creating new files:
some.split.fasta
creating new files:
some.0.fasta
some.1.fasta
creating new files:
some.0
some.1
creating new files:
some.0.1Kmer.fasta
some.1.1Kmer.fasta
creating new files:
some.0.10Kmer.2Koverlap.fasta
some.1.10Kmer.2Koverlap.fasta
creating new files:
some.split.100Kmer.2Koverlap.fasta
................................................................................................................
======================================================================
FAIL: Doctest: pyfasta.fasta.Fasta.sequence
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.7/doctest.py", line 2166, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for pyfasta.fasta.Fasta.sequence
  File "/home/ir210/Envs/reflexdev/src/pyfasta/pyfasta/fasta.py", line 117, in sequence

----------------------------------------------------------------------
File "/home/ir210/Envs/reflexdev/src/pyfasta/pyfasta/fasta.py", line 131, in pyfasta.fasta.Fasta.sequence
Failed example:
    f.sequence({'start':1, 'stop':2, 'strand':1, 'chr': 'chr1'})
Expected:
    'AC'
Got:
    u'AC'
----------------------------------------------------------------------
File "/home/ir210/Envs/reflexdev/src/pyfasta/pyfasta/fasta.py", line 134, in pyfasta.fasta.Fasta.sequence
Failed example:
    f.sequence({'start':1, 'stop':2, 'strand': -1, 'chr': 'chr1'})
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python2.7/doctest.py", line 1254, in __run
        compileflags, 1) in test.globs
      File "<doctest pyfasta.fasta.Fasta.sequence[3]>", line 1, in <module>
        f.sequence({'start':1, 'stop':2, 'strand': -1, 'chr': 'chr1'})
      File "/home/ir210/Envs/reflexdev/src/pyfasta/pyfasta/fasta.py", line 192, in sequence
        sequence = complement(sequence)[::-1]
      File "/home/ir210/Envs/reflexdev/src/pyfasta/pyfasta/fasta.py", line 18, in <lambda>
        complement  = lambda s: s.translate(_complement)
    TypeError: character mapping must return integer, None or unicode
----------------------------------------------------------------------

[brentp]

@ian1roberts , thanks for reporting. can you check the latest commit and see how that works for you? I'll wait to make sure it's OK with python 3 before uploading to pypi.

[ian1roberts]

Thanks so much for fixing it. All 116 pyfasta tests complete without issue, as do the 15 tests from fastinterval - the package that I use that depends on pyfasta

(testpyfasta)[ir210@beast src]$ cd pyfasta/
(testpyfasta)[ir210@beast pyfasta]$ python setup.py nosetests
running nosetests
running egg_info
writing pyfasta.egg-info/PKG-INFO
writing top-level names to pyfasta.egg-info/top_level.txt
writing dependency_links to pyfasta.egg-info/dependency_links.txt
writing entry points to pyfasta.egg-info/entry_points.txt
reading manifest file 'pyfasta.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pyfasta.egg-info/SOURCES.txt'
running build_ext
nose.plugins.cover: ERROR: Coverage not available: unable to import coverage module
....creating new files:
some.split.fasta
creating new files:
some.0.fasta
some.1.fasta
creating new files:
some.0
some.1
creating new files:
some.0.1Kmer.fasta
some.1.1Kmer.fasta
creating new files:
some.0.10Kmer.2Koverlap.fasta
some.1.10Kmer.2Koverlap.fasta
creating new files:
some.split.100Kmer.2Koverlap.fasta
................................................................................................................
----------------------------------------------------------------------
Ran 116 tests in 0.766s

OK

Marking as closed.

All the best
Ian

[brentp]

leaving open til we check on python3

[mruffalo]

Short answer: broken on 3.3.2.

mruffalo@giygas:~/workspace/pyfasta:master$ git rev-parse HEAD
307b030617bda4bf83210ce8302423f672aee0ac
mruffalo@giygas:~/workspace/pyfasta:master$ 2to3-3.3 -w -n . # Mimic what 'setup.py build' does
mruffalo@giygas:~/workspace/pyfasta:master$ python setup.py test
running test
Checking .pth file support in .
/home/mruffalo/.py3-venv/bin/python -E -c pass
Searching for nose
Reading http://pypi.python.org/simple/nose/
Best match: nose 1.3.0
Downloading https://pypi.python.org/packages/source/n/nose/nose-1.3.0.tar.gz#md5=95d6d32b9d6b029c3c65674bd9e7eabe
Processing nose-1.3.0.tar.gz
Writing /tmp/easy_install-gckeog/nose-1.3.0/setup.cfg
Running nose-1.3.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-gckeog/nose-1.3.0/egg-dist-tmp-6w9aef
no previously-included directories found matching 'doc/.build'

Installed /home/mruffalo/workspace/pyfasta/nose-1.3.0-py3.3.egg
running egg_info
creating pyfasta.egg-info
writing top-level names to pyfasta.egg-info/top_level.txt
writing entry points to pyfasta.egg-info/entry_points.txt
writing dependency_links to pyfasta.egg-info/dependency_links.txt
writing pyfasta.egg-info/PKG-INFO
writing manifest file 'pyfasta.egg-info/SOURCES.txt'
reading manifest file 'pyfasta.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pyfasta.egg-info/SOURCES.txt'
running build_ext
/home/mruffalo/workspace/pyfasta/nose-1.3.0-py3.3.egg/nose/config.py:262: RuntimeWarning: Option 'with-doctest' in config file 'setup.cfg' ignored: excluded by runtime environment
  warn(msg, RuntimeWarning)
/home/mruffalo/workspace/pyfasta/nose-1.3.0-py3.3.egg/nose/config.py:262: RuntimeWarning: Option 'doctest-extension' in config file 'setup.cfg' ignored: excluded by runtime environment
  warn(msg, RuntimeWarning)
/home/mruffalo/workspace/pyfasta/nose-1.3.0-py3.3.egg/nose/config.py:262: RuntimeWarning: Option 'with-coverage' in config file 'setup.cfg' ignored: excluded by runtime environment
  warn(msg, RuntimeWarning)
/home/mruffalo/workspace/pyfasta/nose-1.3.0-py3.3.egg/nose/config.py:262: RuntimeWarning: Option 'cover-package' in config file 'setup.cfg' ignored: excluded by runtime environment
  warn(msg, RuntimeWarning)
Failure: AttributeError ('dict' object has no attribute 'decode') ... ERROR
Failure: AttributeError ('dict' object has no attribute 'decode') ... ERROR

======================================================================
ERROR: Failure: AttributeError ('dict' object has no attribute 'decode')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mruffalo/workspace/pyfasta/nose-1.3.0-py3.3.egg/nose/failure.py", line 38, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/home/mruffalo/workspace/pyfasta/nose-1.3.0-py3.3.egg/nose/loader.py", line 413, in loadTestsFromName
    addr.filename, addr.module)
  File "/home/mruffalo/workspace/pyfasta/nose-1.3.0-py3.3.egg/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/mruffalo/workspace/pyfasta/nose-1.3.0-py3.3.egg/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/lib/python3.3/imp.py", line 190, in load_module
    return load_package(name, filename)
  File "/usr/lib/python3.3/imp.py", line 160, in load_package
    return _bootstrap.SourceFileLoader(name, path).load_module(name)
  File "<frozen importlib._bootstrap>", line 584, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1022, in load_module
  File "<frozen importlib._bootstrap>", line 1003, in load_module
  File "<frozen importlib._bootstrap>", line 560, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 868, in _load_module
  File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
  File "/home/mruffalo/workspace/pyfasta/pyfasta/__init__.py", line 3, in <module>
    from .fasta import Fasta, complement, DuplicateHeaderException
  File "/home/mruffalo/workspace/pyfasta/pyfasta/fasta.py", line 17, in <module>
    _complement = maketrans('ATCGatcgNnXx', 'TAGCtagcNnXx').decode('latin-1')
AttributeError: 'dict' object has no attribute 'decode'

======================================================================
ERROR: Failure: AttributeError ('dict' object has no attribute 'decode')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mruffalo/workspace/pyfasta/nose-1.3.0-py3.3.egg/nose/failure.py", line 38, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/home/mruffalo/workspace/pyfasta/nose-1.3.0-py3.3.egg/nose/loader.py", line 413, in loadTestsFromName
    addr.filename, addr.module)
  File "/home/mruffalo/workspace/pyfasta/nose-1.3.0-py3.3.egg/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/mruffalo/workspace/pyfasta/nose-1.3.0-py3.3.egg/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/lib/python3.3/imp.py", line 180, in load_module
    return load_source(name, filename, file)
  File "/usr/lib/python3.3/imp.py", line 119, in load_source
    _LoadSourceCompatibility(name, pathname, file).load_module(name)
  File "<frozen importlib._bootstrap>", line 584, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1022, in load_module
  File "<frozen importlib._bootstrap>", line 1003, in load_module
  File "<frozen importlib._bootstrap>", line 560, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 868, in _load_module
  File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
  File "/home/mruffalo/workspace/pyfasta/tests/test_all.py", line 1, in <module>
    from pyfasta import Fasta
  File "/home/mruffalo/workspace/pyfasta/pyfasta/__init__.py", line 3, in <module>
    from .fasta import Fasta, complement, DuplicateHeaderException
  File "/home/mruffalo/workspace/pyfasta/pyfasta/fasta.py", line 17, in <module>
    _complement = maketrans('ATCGatcgNnXx', 'TAGCtagcNnXx').decode('latin-1')
AttributeError: 'dict' object has no attribute 'decode'

----------------------------------------------------------------------
Ran 2 tests in 0.072s

FAILED (errors=2)

In Python 3, str.maketrans returns a dict since translation tables map Unicode code points to other Unicode code points and there are far more than possible replacements.

I try to avoid checking sys.version_info if possible, but it might be hard to avoid in this case. Something like this should work:

_complement = maketrans('ATCGatcgNnXx', 'TAGCtagcNnXx')
if sys.version_info[0] < 3:
    _complement = _complement.decode('latin-1')
complement  = lambda s: s.translate(_complement)

I haven't had time to test that yet, but I should be able to do so and produce a patch later today.

I also noticed that the doctest changes fail in Python 3 since str.__repr__ doesn't have a leading u like unicode.__repr__ does in 2. It might be necessary to print() the expected values instead of inspecting their __repr__ since 2to3 doesn't touch doctests.