New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recursion error when multiprocessing.Pool.map bam file #20

Closed
olgabot opened this Issue Sep 17, 2018 · 10 comments

Comments

Projects
None yet
3 participants
@olgabot
Copy link

olgabot commented Sep 17, 2018

Describe the bug
Yet another bug to enjoy! This build is failing, but only for python 2.7.

To Reproduce
Steps to reproduce the behavior: Use multiprocessing (or pathos.multiprocessing, a lambda-friendly drop-in replacement for multiprocessing) to spawn multiple processes. Again, uses these bam files

import os

import bamnostic as bs
import pathos.multiprocessing as mp

folder = os.path.join('tests', 'test-data', '10x-example')

bam_file = bs.AlignmentFile(
        os.path.join(folder, 'possorted_genome_bam.bam'), mode='rb')

pool = mp.Pool(processes=2)
pool.map(lambda x: "CB" in x.tags, bam_file)

Expected behavior
Expected to iterate over the alignments in the bam file without error.

Screenshots

  File "/home/travis/build/dib-lab/sourmash/tests/sourmash_tst_utils.py", line 103, in runscript
    status = _runscript(scriptname)
  File "/home/travis/build/dib-lab/sourmash/tests/sourmash_tst_utils.py", line 50, in _runscript
    pkg_resources.load_entry_point("sourmash", 'console_scripts', scriptname)()
  File "/home/travis/build/dib-lab/sourmash/sourmash/__main__.py", line 77, in main
    cmd(sys.argv[2:])
  File "/home/travis/build/dib-lab/sourmash/sourmash/commands.py", line 278, in compute
    pool.map(lambda x: maybe_add_alignment(x, cell_seqs, args, barcodes), bam_file)
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/multiprocess/pool.py", line 253, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/multiprocess/pool.py", line 306, in map_async
    iterable = list(iterable)
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/bgzf.py", line 1420, in next
    read = bamnostic.AlignedSegment(self)
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/core.py", line 199, in __init__
    bsize_buffer = self._io.read(4)
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/bgzf.py", line 832, in read
    self._load_block()  # will reset offsets
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/bgzf.py", line 535, in _load_block
    self._buffers[self._block_start_offset] = self._buffer, block_size
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/utils.py", line 516, in __setitem__
    self.cull()
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/utils.py", line 486, in cull
    self.popitem(last=False)
  File "/opt/python/2.7.14/lib/python2.7/collections.py", line 178, in popitem
    value = self.pop(key)
  File "/opt/python/2.7.14/lib/python2.7/collections.py", line 156, in pop
    result = self[key]
=== begin repeated many times ===
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/utils.py", line 500, in __getitem__
    self[key] = value
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/utils.py", line 516, in __setitem__
    self.cull()
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/utils.py", line 486, in cull
    self.popitem(last=False)
  File "/opt/python/2.7.14/lib/python2.7/collections.py", line 178, in popitem
    value = self.pop(key)
  File "/opt/python/2.7.14/lib/python2.7/collections.py", line 156, in pop
    result = self[key]
=== end repeated many times ===
  File "/home/travis/build/dib-lab/sourmash/.tox/py27/lib/python2.7/site-packages/bamnostic/utils.py", line 498, in __getitem__
    if not key == list(self.keys())[-1]:
  File "/opt/python/2.7.14/lib/python2.7/collections.py", line 119, in keys
    return list(self)
RuntimeError: maximum recursion depth exceeded while calling a Python object

Desktop (please complete the following information):

  • OS: macOS
  • Python Version: 2.7
  • bamnostic Version: v0.9.1

Additional context
None

@betteridiot

This comment has been minimized.

Copy link
Owner

betteridiot commented Sep 17, 2018

Thank you again for making me aware of this. I had thought I fixed the recursion problem in 2.7, but it looks like I will have to take another look at it. Should have it fixed sometime tomorrow morning.

@betteridiot

This comment has been minimized.

Copy link
Owner

betteridiot commented Sep 18, 2018

Okay...ugh, that was painful. v0.9.1rc3 should fix the recursion error on Python 2.7

I had to rewrite the way py27 OrderedDict works, and add the move_to_end method into it. There is also a minor change to bamnostic.bgzf to reflect the new api to bamnostic.utils.LruDict

@betteridiot

This comment has been minimized.

Copy link
Owner

betteridiot commented Sep 18, 2018

Please keep up the good work

@betteridiot betteridiot reopened this Sep 18, 2018

@betteridiot

This comment has been minimized.

Copy link
Owner

betteridiot commented Sep 18, 2018

Nevermind...now my build is failing on tuple comparisons

@olgabot

This comment has been minimized.

Copy link
Author

olgabot commented Sep 18, 2018

Haha I'm glad to help! Really appreciate the speed at which you're fixing these errors.

@betteridiot

This comment has been minimized.

Copy link
Owner

betteridiot commented Sep 18, 2018

Finally done. v0.9.1rc4 has the newest release with recursion error corrections

@luizirber

This comment has been minimized.

Copy link

luizirber commented Sep 18, 2018

You're both doing a stellar job =]

@betteridiot

This comment has been minimized.

Copy link
Owner

betteridiot commented Sep 18, 2018

BTW, thanks @luizirber for the recommendation you gave to @olgabot to look at bamnostic. She is really using the package and helping me find the holes in it.

@betteridiot

This comment has been minimized.

Copy link
Owner

betteridiot commented Sep 18, 2018

v0.9.2 is out. This doesn't change anything of value except ensuring versioning matches that of PyPI and conda

@olgabot

This comment has been minimized.

Copy link
Author

olgabot commented Sep 19, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment