No seq file generated

[smazrouee]

Below is a sample config file where we don't get the sequences (/subsamples):

{
"ContactNetwork": "NetworkX",
"ContactNetworkGenerator": "RandomPartitionGraph",
"d_or_u": "u",

"rpg_p_in": 0.004,
"rpg_p_out": 0.0002,
"rpg_sizes": [1000,1000,1000,1000,1000],
"end_time": 10,
"num_seeds": 50,
"out_dir": "$FAVITESHOME/output/set6/output_6_19",

"Driver": "Default",
"EndCriteria": "GEMF",
"Logging": "File",

"NodeAvailability": "Bernoulli",
"node_sample_prob": 1,
"NumBranchSample": "Single",
"NumTimeSample": "Once",
"SourceSample": "Random",
"TimeSample": "GranichFirstART",

"NodeEvolution": "VirusTreeSimulator",
"vts_growthRate": 2.851904,
"vts_model": "logistic",
"vts_n0": 1,
"vts_t50": -2,

"PostValidation": "Dummy",

"SeedSelection": "Random",

"SeedSequence": "VirusPureCoalescentGTRGamma",
"viral_sequence_type": "HIV1-B-DNA-POL-LITTLE",

"SequenceEvolution": "GTRGammaSeqGen",
"Sequencing": "Perfect",

"TransmissionNodeSample": "GEMF",
"hiv_a1_to_d": 0,
"hiv_a2_to_a3": 0,
"hiv_a2_to_d": 0,
"hiv_a1_to_i1": 0.48,
"hiv_a2_to_i2": 0.48,
"hiv_i2_to_a2": 0.5,
"hiv_i1_to_a1": 0.5,
"hiv_a3_to_a4": 0,
"hiv_a3_to_d": 0,
"hiv_a3_to_i3": 0,
"hiv_a4_to_d": 0,
"hiv_a4_to_i4": 0,
"hiv_i1_to_d": 0,
"hiv_i1_to_i2": 8.666666666666666,
"hiv_a1_to_a2": 4.33333,
"hiv_i2_to_d": 0,
"hiv_i2_to_i3": 0,
"hiv_i3_to_a3": 0,
"hiv_i3_to_d": 0,
"hiv_i3_to_i4": 0,
"hiv_i4_to_a4": 0,
"hiv_i4_to_d": 0,
"hiv_ns_to_d": 0,
"hiv_ns_to_s": 999999,
"hiv_s_to_d": 0,
"hiv_s_to_i1_by_a1": 0.005625,
"hiv_s_to_i1_by_a2": 0,
"hiv_s_to_i1_by_a3": 0,
"hiv_s_to_i1_by_a4": 0,
"hiv_s_to_i1_by_i1": 0.1125,
"hiv_s_to_i1_by_i2": 0.0225,
"hiv_s_to_i1_by_i3": 0,
"hiv_s_to_i1_by_i4": 0,
"hiv_s_to_i1_seed": 0,
"TransmissionTimeSample": "HIVARTGranichGEMF",

"TreeNode": "Simple",
"TreeUnit": "LogNormal",
"tree_rate_mean": -6.164,
"tree_rate_stdev": 0.3,

"grinder_path": "grinder",
"gemf_path": "GEMF",
"hmmemit_path": "hmmemit",
"java_path": "java",
"nw_rename_path": "nw_rename",
"seqgen_path": "seq-gen",

"seed_population": 20,

"seqgen_a_to_c": 1.765707,
"seqgen_a_to_g": 9.587649,
"seqgen_a_to_t": 0.691915,
"seqgen_c_to_g": 0.863348,
"seqgen_c_to_t": 10.282617,
"seqgen_freq_a": 0.392,
"seqgen_freq_c": 0.165,
"seqgen_freq_g": 0.212,
"seqgen_freq_t": 0.232,
"seqgen_g_to_t": 1.0,
"seqgen_gamma_shape": 0.405129,
"seqgen_num_gamma_rate_categories": ""

}

[niemasd]

What version are you using? I just ran it 20 times and got sequences each time (both error-free and error-prone)

[smazrouee]

This is the problem which makes it hard for me to trace and find the reason. I also ran the same Config file 10-20 times and only sometimes it doesn't generate sequences, while the config is exactly the same (except the out_dir).

I have more than one example with this problem, but here couldn't upload. So I copied the content of one file. This run is the latest version from Feb 28th btw (before you incorporate -u)

[niemasd]

I ran it 150 times just now, and it succeeded and produced sequences every time. Perhaps try using the most recent versioned FAVITES Docker image (via -u) and see if it still happens?

[smazrouee]

(probably) because of stochasticity incorporated in the software you are unable to get the same exact results yet (or the chance of bumping to the same problem is next to zero). Sounds to me it's hard to trace back and see what was the reason, which is understandable... Maybe later (after you implemented reproducibility) you can come back and look at this

[niemasd]

(probably) because of stochasticity incorporated in the software you are unable to get the same exact results yet (or the chance of bumping to the same problem is next to zero)

I would think this is the case, but you earlier said the following:

I also ran the same Config file 10-20 times and only sometimes it doesn't generate sequences

This implies you are running into this multiple times in only 10-20 executions, whereas I ran it 150 times and didn't run into it at all. Even assuming you only saw it once out of 20 times (so roughly Geometric distribution with p = 0.05), me not observing it after 150 tries has probability (1-0.5)^150 = 0.0005. Thus, I don't think it's likely that stochasticity is the reason why I'm not able to reproduce the issue

[smirarab]

Niema, are you running with docker? Perhaps runs with Docker are a little bit less stable for some reason? Can you try with Docker as well?

…

On Mon, Mar 12, 2018 at 3:33 PM, Niema Moshiri ***@***.***> wrote: (probably) because of stochasticity incorporated in the software you are unable to get the same exact results yet *(or the chance of bumping to the same problem is next to zero)* I would think this is the case, but you earlier said the following: I also ran the same Config file 10-20 times and only *sometimes* it doesn't generate sequences This implies you are running into this multiple times in only 10-20 executions, whereas I ran it 150 times and didn't run into it at all. Even assuming you only saw it once out of 20 times (so roughly Geometric distribution with *p* = 0.05), me not observing it after 150 tries has probability (1-0.5)^150 = 0.0005. Thus, I don't think it's likely that stochasticity is the reason why I'm not able to reproduce the issue — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAybuG3VDwQ1rjAhejxkl8nhOSqCg08Dks5tdvfLgaJpZM4SnRVi> .

-- Siavash Mirarab

[niemasd]

I just ran it 150 times using the 1.0.3 Docker image on Calab, and I had the same result (sequences exist each time). Here is how I'm running it:

r=1; while : ; do echo $r && ((r++)) && rm -rf test && ~/bin/FAVITES/run_favites_docker.py -c test.json -o test && ls test/error_free_files/sequence_data.fasta > /dev/null || break; done

Basically, I have an infinite loop (while : ; do ... ; done) where I run FAVITES via run_favites_docker.py (with the 1.0.3 image in my environment) on the config file (I'm having it output to a directory called test for simplicity), and I attempt to ls the file test/error_free_files/sequence_data.fasta. If this succeeds (i.e., the file exists), the logical OR (||) does not enter the second condition (break), so the loop continues. If this fails (i.e., the file does not exist), the first condition of the logical OR fails, so it enters the second condition (break) and terminates the loop. Thus, as long as the loop doesn't terminate, FAVITES is running successfully. I just arbitrarily killed it once I hit 150.

[niemasd]

I'll close this for now because I'm unable to reproduce the issue using the latest versioned Docker image (1.0.3), so if it was a bug in FAVITES, I assume it's been fixed