Segmentation fault during larger parallel alignment jobs

[valentynbez]

During a larger alignment job, my kernel dies. I attach the dataset I used in ZIP file:
pickles.zip

Reproducible snippet:

import pyopal 
import pickle
from multiprocessing.pool import ThreadPool

target_sequences = pickle.load(open("target_seqs.pkl", "rb"))
query_sequences = pickle.load(open("query_seqs.pkl", "rb"))
database = pyopal.Database(target_sequences)

with ThreadPool(8) as pool:
    hits = dict(pool.map(lambda q: (q, database.search(q, algorithm="nw", mode="full")), query_sequences))

pyopal version: 0.2.0

EDIT: tried overflow="buckets", no luck.

[althonos]

Looks like an Opal bug:

$ ./opal_aligner -a NW -x 2 /tmp/query.faa /tmp/targets.faa 
Using NW alignment mode.
Reading query fasta file...
Read query sequence, 311 residues.

Reading database fasta file...
Read 131817 database sequences, 41551837 residues total.
Whole database read: 131817 database sequences, 41551837 residues in total.

Comparing query to database...fish: Job 1, './opal_aligner -a NW -x 2 /tmp/…' terminated by signal SIGSEGV (Address boundary error)

It doesn't happen with the Smith-Waterman algorithm (-a SW -x 2), and doesn't happen in partial alignment mode (-a NW -x 1) so this looks like a bug specific to Needleman-Wunsch and full alignment (-a NW -x 2).

[althonos]

Should be fixed in v0.3.0.