halTreePhyloP.py failed w/ unclear error

[brantfaircloth]

Good morning,

After taking alternate steps to generate the model file, I was running halTreePhyloP.py against a large alignment of bird taxa (extracted from a larger alignment of vertebrate taxa), and I hit the following error (using the singularity container):

% singularity exec -B /mnt/data1/bcf /home/bcf/containers/cactus-v2.8.2.sif halTreePhyloP.py --root BirdsAnc0 --numProc 24 --bigWig vgp-577way-v1.BirdsAnc0.hal GCF_016700215.2.HKY85.mod vgp-577way-phylop

HDF5-DIAG: Error detected in HDF5 (1.10.7) thread 1:
  #000: ../../../src/H5Dio.c line 184 in H5Dread(): could not get a validated dataspace from file_space_id
    major: Invalid arguments to routine
    minor: Bad value
  #001: ../../../src/H5S.c line 257 in H5S_get_validated_dataspace(): selection + offset not within extent
    major: Dataspace
    minor: Out of range
terminate called after throwing an instance of 'H5::DataSetIException'
Aborted (core dumped)
Traceback (most recent call last):
  File "/home/cactus/bin/halTreePhyloP.py", line 157, in <module>
    sys.exit(main())
  File "/home/cactus/bin/halTreePhyloP.py", line 154, in main
    computeTreePhyloP(args)
  File "/home/cactus/bin/halTreePhyloP.py", line 74, in computeTreePhyloP
    runShellCommand("halWiggleLiftover %s %s %s %s %s --append" % (
  File "/home/cactus/hal/stats/halStats.py", line 27, in runShellCommand
    raise RuntimeError("Command: %s exited with non-zero status %i" %
RuntimeError: Command: halWiggleLiftover vgp-577way-v1.BirdsAnc0.hal BirdsAnc0 vgp-577way-phylop/BirdsAnc0_phyloP.wig PalaeognathaeAnc0 vgp-577way-phylop/PalaeognathaeAnc0_phyloP.wig --append exited with non-zero status 134

Given the error, this might be recoverable (??) - there is a large (6.6G) wiggle file (BirdsAnc0_phyloP.wig) that was produced and output by the above, before the attempt to liftover failed.

[glennhickey]

Yeah, not much to go on with that error. Could have run out of memory? Unfortunately, I don't see an easy path to recovery.

I really want to re-emphasize my previous comment: the phast integration in HAL hasn't been super well-maintained, nor (to my knowledged) tested at these scales. It would be nice to bring it up to speed, but in the meantime probably best to compute conservation directly on the MAF(s) via the latest phast tooling.

[brantfaircloth]

Sounds good - I'm on it (relative to using phast). I was hoping for a quick workaround...

[glennhickey]

Looking at this halTreePhyloP sceript (it's been ages).. I think the simplest and safest way to bring this up to to date would be:

• halPhyloPMP.py: Add Toil support, take MAF input (or generate on the fly with cactus-hal2maf toil logic if needed). From their a toil-distributed workflow to compute phyloP from the MAF shards with latest and greatest multithreaded phylop
• halTreePhyloP: Add Toil support; use toil-based phylop workflow above.

This doesn't seem like a ton of development, but isn't something I can personally get to right away.