Recover/retry on alignment thread death

[mhgislason]

One thread died during alignment of DeepM6ASeq. It would be nice if a retry was implemented instead of complete failure.

[maghal@i-02-c0157 graph_part]$ python graph_part.py needle --fasta-file ../benchmarking/data/rna/DeepM6ASeq/deepm6aseq_dr.fasta --threshold 0.2 --out-file ../../deepm6seq_partitions_020_labels.csv --labels-name label --partitions 5 --save-checkpoint-path ../../deepm6seq_default_th020_labels.edges --threads 22 --chunks 22
{'time_script_start': 5003792.789758933, 'config': {'alignment_mode': 'needle', 'fasta_file': '../benchmarking/data/rna/DeepM6ASeq/deepm6aseq_dr.fasta', 'threshold': 0.2, 'partitions': 5, 'transformation': 'one-minus', 'out_file': '../../deepm6seq_partitions_020_labels.csv', 'priority_name': None, 'labels_name': 'label', 'initialization_mode': 'slow-nn', 'aggressivity': 8, 'skip_removal': 0, 'no_moving': False, 'remove_same': False, 'load_checkpoint_path': None, 'save_checkpoint_path': '../../deepm6seq_default_th020_labels.edges', 'denominator': 'full', 'nucleotide': False, 'triangular': False, 'threads': 22, 'chunks': 22, 'gapopen': 10, 'gapextend': 0.5, 'endweight': False, 'endopen': 10, 'endextend': 10, 'matrix': 'EBLOSUM62', 'threshold_transformed': 0.8}}
val num lim
DR_NEG 0 13878 2775
DR_POS 1 13881 2776
Computing pairwise sequence identities.
2%|████▏ | 13840606/770562081 [05:33<6:17:00, 33453.01it/s] 45%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 345429826/770562081 [3:24:27<22:24:58, 5268.16it/s]not enough values to unpack (expected 2, got 1)
Traceback (most recent call last):
File "graph_part.py", line 631, in
main()
File "graph_part.py", line 509, in main
matrix=args.matrix )
File "/ngc/projects/gm/people/maghal/projects/gp-test/graph-part/graph_part/needle_utils.py", line 354, in generate_edges_mp
raise RuntimeError('One of the alignment threads did not complete sucessfully.')
RuntimeError: One of the alignment threads did not complete sucessfully.