possible mpi bug w/ SLAC

[sjspielman]

Hi @spond (and @stevenweaver, as this may influence 2.3.7 release timing!),

I believe I have uncovered a SLAC MPI bug:

Stephanies-MacBook-Pro:phyphy sjspielman$ mpirun -np 3 /Users/sjspielman/Desktop/bin/HYPHYMPI LIBPATH=/Users/sjspielman/Desktop/lib/hyphy/ /Users/sjspielman/Desktop/lib/hyphy/TemplateBatchFiles/SelectionAnalyses/SLAC.bf Universal /Users/sjspielman/simdata/codon.fna Y All 100 0.1

Analysis Description
--------------------
SLAC (Single Likelihood Ancestor Counting) uses a maximum likelihood
ancestral state reconstruction and minimum path substitution counting to
estimate site - level dS and dN, and applies a simple binomial - based
test to test if dS differs drom dN. The estimates aggregate information
over all branches, so the signal is derived from pervasive
diversification or conservation. A subset of branches can be selected
for testing as well. Multiple partitions within a NEXUS file are also
supported for recombination - aware analysis. 

- __Requirements__: in-frame codon alignment and a phylogenetic tree

- __Citation__: Not So Different After All: A Comparison of Methods for Detecting Amino
Acid Sites Under Selection (2005). _Mol Biol Evol_ 22 (5): 1208-1222

- __Written by__: Sergei L Kosakovsky Pond and Simon DW Frost

- __Contact Information__: spond@temple.edu

- __Analysis Version__: 2.00



>A tree was found in the data file: `((((t8:0.01,(t13:0.07,t4{stephanie}:0.18):0.06):0.24,(t1:0.14,(t7:0.11,t15:0.03):0.21):0.11):0.09,(t5:0.05,t14{steph}:0.31):0.111):0.15,(((t9:0.04,t6:0.15):0.46,(t12:0.10,t11:0.12):0.25):0.17,(t2:0.04,(t3:0.05,t10:0.19):0.0):0.29):0.27)`

>Would you like to use it (y/n)? Y


>Loaded a multiple sequence alignment with **15** sequences, **10** codons, and **1** partitions from `/Users/sjspielman/Research/phyphy/tests/test_data/codon.fna`

>Select the number of samples used to assess ancestral reconstruction uncertainty [select 0 to skip] (permissible range = [0,100000], default value = 100, integer): 100

>Select the p-value threshold to use when testing for selection (permissible range = [0,1], default value = 0.1): 0.1


### Branches to include in the SLAC analysis
Selected 27 branches to include in SLAC calculations: `t8, t13, t4, Node5, Node3, t1, t7, t15, Node10, Node8, Node2, t5, t14, Node13, t9, t6, Node18, t12, t11, Node21, Node17, t2, t3, t10, Node26, Node24, Node16`


### Obtaining branch lengths and nucleotide substitution biases under the nucleotide GTR model
* Log(L) =  -197.18, AIC-c =   470.45 (35 estimated parameters)

### Obtaining the global omega estimate based on relative GTR branch lengths and nucleotide substitution biases
* Log(L) =  -189.74, AIC-c =   497.24 (42 estimated parameters)
* non-synonymous/synonymous rate ratio for *test* =   0.4329

### Performing joint maximum likelihood ancestral state reconstruction

### For partition 1 these sites are significant at p <=0.1

|     Codon      |   Partition    |       S        |       N        |       dS       |       dN       |Selection detected?|
|:--------------:|:--------------:|:--------------:|:--------------:|:--------------:|:--------------:|:-----------------:|
|       6        |       1        |     4.000      |     0.000      |     3.804      |     0.000      |  Neg. p = 0.015   |
|       9        |       1        |     4.000      |     1.000      |     3.927      |     0.505      |  Neg. p = 0.048   |

### Ancestor sampling analysis

>Generating 100 ancestral sequence samples to obtain confidence intervals

	Sample 3/100 for partition 1HYPHYMPI terminated.
Error:
Failed to dereference 'terms.genetic_code.EPS'

Function call stack
1 : A return statement with:1
-------
2 : pairwise_eps=counts[utility.getGlobalValue("terms.genetic_code.EPS")]
-------
3 : slac.sampled=ancestral.build(lf,partition,{"sample":TRUE})
-------
4 : A return statement with:-1
-------


--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
Error:

Master node received an error:HyPhy killed by signal 15

Function call stack
1 : MPI Receive from -1 storing actual sender node into from and storing the string result into result
	Standard input redirect:
		Empty Associative List-------
2 : node=aux._handle_receieve(queue)
	Standard input redirect:
		Empty Associative List-------
3 : mpi.QueueJob(slac.queue,"slac.handle_a_sample",{"0":slac.partitioned_mg_results[terms.likelihood_function],"1":slac.i,"2":slac.selected_branches[slac.i],"3":slac.counts},"slac.handle_a_sample_callback")
	Standard input redirect:
		Empty Associative List-------
HYPHYMPI terminated.
Error:
HyPhy killed by signal 15

Function call stack
1 : _bac_tree_avl=(^_bac_treeID)^0
-------
2 : A return statement with:1
-------
3 : slac.sampled=ancestral.build(lf,partition,{"sample":TRUE})
-------
4 : A return statement with:-1
-------

This seems SLAC specific (likely because only SLAC is using the relevant code). I'm attaching a dataset to reproduce and log files.
slacbug.zip
Run w/ current beta. Thanks!

-Stephanie

[spond]

Dear @sjspielman,

Easy fix. Missing headers and variables when setting up the MPI queue.

Best,
Sergei