something up in cactus_createMultiProject.py

[benedictpaten]

Running cactus_createMultiCactusProject.py (using the latest development branch of cactus) I got the following error:

[benedict@kolossus progCactusError]$ cactus_createMultiCactusProject.py /cluster/home/benedict/progCactusError/experiment.xml ./progressiveCactusAlignment --fixNames=False --rootOutgroupPath /hive/users/benedict/datasets/realMammals/hg19.fa --rootOutgroupDist 1.0
Traceback (most recent call last):
File "/hive/users/benedict/cactus/bin/cactus_createMultiCactusProject.py", line 234, in
main()
File "/hive/users/benedict/cactus/bin/cactus_createMultiCactusProject.py", line 228, in main
createFileStructure(mcProj, expTemplate, confTemplate, options)
File "/hive/users/benedict/cactus/bin/cactus_createMultiCactusProject.py", line 127, in createFileStructure
exp.updateTree(subtree, seqMap)
File "/hive/users/benedict/cactus/shared/experimentWrapper.py", line 416, in updateTree
sequences += seqMap[nodeName]
KeyError: 'fishAnc00'

I've put the example in /cluster/home/benedict/progCactusError/

I think this relates to Joel's latest changes, so I've assigned to him. I will look again when I get a chance.

[joelarmstrong]

I think this relates to the fishAnc00:0.326688; in the tree, i.e. the root node shouldn't have a branch length.

This may be my fault -- I think I was the one to give you the tree.

[benedictpaten]

Good point, but we should have a more informative error message.

Benedict (from phone)
On Jan 23, 2014 10:39 AM, "Joel Armstrong" notifications@github.com wrote:

I think this relates to the fishAnc00:0.326688; in the tree, i.e. the root
node shouldn't have a branch length.

This may be my fault -- I think I was the one to give you the tree.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5#issuecomment-33154204
.

[benedictpaten]

So I removed the root branch length, but then I got the error:

Reporting file:
/hive/users/benedict/progressiveBenchmarks/output/lotsOfFish_rm10_ogGreedy_ot0_saFalse_ktInmemorynosnapshot/tempProgressiveCactusAlignment/jobTree/jobs/log.txt

log.txt: log.txt: ---JOBTREE SLAVE OUTPUT LOG---

log.txt: log.txt: Traceback (most recent call last):

log.txt: log.txt: File
"/hive/users/benedict/jobTree/src/jobTreeSlave.py", line 271, in main

log.txt: log.txt: defaultMemory=defaultMemory, defaultCpu=defaultCpu,
depth=depth)

log.txt: log.txt: File
"/hive/users/benedict/jobTree/scriptTree/stack.py", line 153, in execute

log.txt: log.txt: self.target.run()

log.txt: log.txt: File
"/hive/users/benedict/cactus/progressive/cactus_progressive.py", line 200,
in run

log.txt: log.txt: assert self.options.event in project.expMap

log.txt: log.txt: AssertionError

log.txt: log.txt: Exiting the slave because of a failed job on host kolossus

log.txt: log.txt: Due to failure we are reducing the remaining retry count
of job
/hive/users/benedict/progressiveBenchmarks/output/lotsOfFish_rm10_ogGreedy_ot0_saFalse_ktInmemorynosnapshot/tempProgressiveCactusAlignment/jobTree/jobs/job
to 2

log.txt: log.txt: We have set the default memory of the failed job to
8589934593.0 bytes

log.txt: The job seems to have left a log file, indicating failure:
/hive/users/benedict/progressiveBenchmarks/output/lotsOfFish_rm10_ogGreedy_ot0_saFalse_ktInmemorynosnapshot/tempProgressiveCactusAlignment/jobTree/jobs/job

log.txt: Reporting file:
/hive/users/benedict/progressiveBenchmarks/output/lotsOfFish_rm10_ogGreedy_ot0_saFalse_ktInmemorynosnapshot/tempProgressiveCactusAlignment/jobTree/jobs/log.txt

On Thu, Jan 23, 2014 at 10:44 AM, Benedict Paten benedict@soe.ucsc.eduwrote:

Good point, but we should have a more informative error message.

Benedict (from phone)
On Jan 23, 2014 10:39 AM, "Joel Armstrong" notifications@github.com
wrote:

I think this relates to the fishAnc00:0.326688; in the tree, i.e. the
root node shouldn't have a branch length.

This may be my fault -- I think I was the one to give you the tree.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5#issuecomment-33154204
.

[joelarmstrong]

Did you delete the jobTree? It looks like it has references to "Anc00" (the node that accidentally got created) in the pickles.

[benedictpaten]

Oops. Are you saying we can't name root branches at all?

On Thu, Jan 23, 2014 at 12:03 PM, Joel Armstrong
notifications@github.comwrote:

Did you delete the jobTree? It looks like it has references to "Anc00"
(the node that accidentally got created) in the pickles.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5#issuecomment-33162646
.

[benedictpaten]

Again, a message to that tune would be good! :)

On Thu, Jan 23, 2014 at 12:58 PM, Benedict Paten benedict@soe.ucsc.eduwrote:

Oops. Are you saying we can't name root branches at all?

On Thu, Jan 23, 2014 at 12:03 PM, Joel Armstrong <notifications@github.com

wrote:

Did you delete the jobTree? It looks like it has references to "Anc00"
(the node that accidentally got created) in the pickles.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5#issuecomment-33162646
.

[joelarmstrong]

You can name root branches just fine, but the tree I originally gave you (the one with the branch length for the root node) ends up creating a tree with a node above fishAnc00 when it's parsed. It's probably labeled Anc00 since that's the default for unlabeled nodes.

Fixing the tree fixed the source of the problem, but there were still some copies of the old tree in the jobTree pickles. (The tree gets passed around as a parameter to all the cactus_progressive.py targets.) Fixing the tree-string in the experiment file changed the tree topology, but since some of the pickles were inconsistent with the new topology you got weird errors.

Yeah, this is pretty non-obvious--I think we should fail on parsing a malformed newick tree like the one I gave you. I'll push a fix sometime soon.

[joelarmstrong]

This was fixed a while ago.