checkm analyze error

[snayfach]

Hi Donovan,

I'm getting an error when running CheckM v1.0.7.

I've split up my genomes into batches of 250. Each batch is run on a computing cluster with multiple threads. Most batches complete without error. However, for some get an error at the step checkm analyze. All previous steps complete without error. Here's the error:

Finished processing 5 of 250 (2.00%) bins.^M  [Error] Make sure HMMER executables (e.g., hmmsearch, hmmfetch) are on yo
ur system path.
  [Error] Make sure HMMER executables (e.g., hmmsearch, hmmfetch) are on your system path.
Process Process-47:
Traceback (most recent call last):
  File "/global/homes/s/snayfach/.conda/envs/checkm/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/global/homes/s/snayfach/.conda/envs/checkm/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/global/homes/s/snayfach/.conda/envs/checkm/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/global/homes/s/snayfach/.conda/envs/checkm/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/global/homes/s/snayfach/.conda/envs/checkm/lib/python2.7/site-packages/checkm/hmmerAligner.py", line 199, in __c
reateMSA
    self.__alignMarker(markerId, markersWithMultipleHits[markerId], None, False, binAlignOutputDir, tempModelFile, bKeepUnmaskedAlign=False)
  File "/global/homes/s/snayfach/.conda/envs/checkm/lib/python2.7/site-packages/checkm/hmmerAligner.py", line 296, in __alignMarker
    self.__maskAlignment(alignSeqFile, makedSeqFile)
  File "/global/homes/s/snayfach/.conda/envs/checkm/lib/python2.7/site-packages/checkm/hmmerAligner.py", line 355, in __maskAlignment
    maskedSeq = ''.join([seq[i] for i in xrange(0, len(seq)) if mask[i] == 'x'])
UnboundLocalError: local variable 'mask' referenced before assignment
  [Error] Make sure HMMER executables (e.g., hmmsearch, hmmfetch) are on your system path.
  [Error] Make sure HMMER executables (e.g., hmmsearch, hmmfetch) are on your system path.
Process Process-49:
Traceback (most recent call last):
  File "/global/homes/s/snayfach/.conda/envs/checkm/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/global/homes/s/snayfach/.conda/envs/checkm/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/global/homes/s/snayfach/.conda/envs/checkm/lib/python2.7/site-packages/checkm/hmmerAligner.py", line 199, in __createMSA
    self.__alignMarker(markerId, markersWithMultipleHits[markerId], None, False, binAlignOutputDir, tempModelFile, bKeepUnmaskedAlign=False)
  File "/global/homes/s/snayfach/.conda/envs/checkm/lib/python2.7/site-packages/checkm/hmmerAligner.py", line 296, in __alignMarker
    self.__maskAlignment(alignSeqFile, makedSeqFile)
  File "/global/homes/s/snayfach/.conda/envs/checkm/lib/python2.7/site-packages/checkm/hmmerAligner.py", line 331, in __maskAlignment
    for line in open(inputFile):
IOError: [Errno 2] No such file or directory: '/global/projectb/scratch/snayfach/projects/gut_mags/4_checkm/checkm_out/mags/217/checkm/storage/aai_qa/RheumatoidArthritis_D188.012/PF02367.12.aligned.faa'

Unexpected error: <type 'exceptions.AttributeError'>
Traceback (most recent call last):
  File "/global/homes/s/snayfach/.conda/envs/checkm/bin/checkm", line 712, in <module>
    checkmParser.parseOptions(args)
  File "/global/homes/s/snayfach/.conda/envs/checkm/lib/python2.7/site-packages/checkm/main.py", line 1233, in parseOptions
    self.analyze(options)
  File "/global/homes/s/snayfach/.conda/envs/checkm/lib/python2.7/site-packages/checkm/main.py", line 331, in analyze
    os.path.join(options.out_folder, 'storage', 'aai_qa')
  File "/global/homes/s/snayfach/.conda/envs/checkm/lib/python2.7/site-packages/checkm/hmmerAligner.py", line 175, in makeAlignmentsOfMultipleHits
    p.terminate()
  File "/global/homes/s/snayfach/.conda/envs/checkm/lib/python2.7/multiprocessing/process.py", line 137, in terminate
    self._popen.terminate()
AttributeError: 'NoneType' object has no attribute 'terminate'

I've run one of these failed batches of 250 genomes interactively and I get the same error.

I took the same failed batch, and ran each genome individually through CheckM (250 CheckM runs), each with a single thread, and did not get any error.

Any help/ideas would be appreciated.

Thanks,
Stephen

[donovan-h-parks]

Hey Stephen,

For the error log, it would appear you don't have HMMER on your path:
[Error] Make sure HMMER executables (e.g., hmmsearch, hmmfetch) are on your system path.

However, since you have successfully run other batches and individual genomes this seems odd. Can you confirm HMMER >=3.1b1 is on your path when doing this batch run?

Cheers,
Donovan

[snayfach]

Yes, I can confirm that HMMER is on my PATH.

The error message reads:
IOError: [Errno 2] No such file or directory: 'checkm/storage/aai_qa/RheumatoidArthritis_D188.012/PF02367.12.aligned.faa'

And indeed this file does not exist:

$ ls checkm/storage/aai_qa/RheumatoidArthritis_D188.012
PF02367.12.unaligned.faa  TIGR03725.masked.faa

But this file does exist for other bins in the same batch, indicating that this step is failing only for certain bins:

$ ls checkm/storage/aai_qa/RheumatoidArthritis_D191_M.040:
PF00297.17.masked.faa  PF00410.14.masked.faa  PF00410.14.unaligned.faa  PF00673.16.aligned.faa  PF00673.16.unaligned.faa

But as I mentioned before, I can run each bin individually without error. But that doesn't explain why I only get this error for certain batches of genomes and not for others.

Any ideas?

[snayfach]

And more specifically, HMMER 3.1b1 is on my path

[snayfach]

Here is the final output for the error-causing bin when I run it on its own:

Bin Id  Marker lineage  # genomes       # markers       # marker sets   0       1       2       3       4       5+      Completeness    Contamination   Strain heterogeneity
RheumatoidArthritis_D188.012    o__Clostridiales (UID1212)      172     261     147     52      207     2       0       0       0       96.04   0.68    0.00

[snayfach]

I reran Checkm using a reduced # of threads (from all available CPUs to 10). This solved the issue. It's still not totally clear what was happening.

[donovan-h-parks]

I haven't run across this issue before. Do you have other bins with the prefix "RheumatoidArthritis_D188"?
I'm wondering if the ".012" is being interpreted as a file extension somewhere in the code and thus "RheumatoidArthritis_D188.xxx" bins might be clashing at some point. I am happy to dig further, but I think you would need to send me a subset of bins that reproduces the problem (or put all 250 on an FTP site for me).

[donovan-h-parks]

Marking as resolved as version number could not be verified and issue has not been reported by others.

[eskirton]

I've also received this error (with v1.0.7) and similarly reducing the number of threads solved the issue.
I've run CheckM on thousands of metagenomes successfully, usually with 8 threads and 48GB RAM on the cluster, so it's normally working fine. And I have run it with 40 threads successfully (on different assemblies) before too. At this time I'm not sure what the issue is but I'll dig into it some more and get back to you. At the very least, I can send you a dataset so you can reproduce the error yourself (give me a week or two).
The error follows:

---

[CheckM - tetra] Calculating tetranucleotide signature of sequences.

---

Determining tetranucleotide signature of each sequence.
Finished processing 75 of 7266222 (0.00%) sequences.
Unexpected error: <type 'exceptions.AttributeError'>
Traceback (most recent call last):
File "/global/dna/projectdirs/MEP/tools/python/bin/checkm", line 712, in
checkmParser.parseOptions(args)
File "/global/dna/projectdirs/MEP/tools/python/lib/python2.7/site-packages/checkm/main.py", line 1289, in parseOptions
self.tetraSignatures(options)
File "/global/dna/projectdirs/MEP/tools/python/lib/python2.7/site-packages/checkm/main.py", line 914, in tetraSignatures
tetraSig.calculate(options.seq_file, options.output_file)
File "/global/dna/projectdirs/MEP/tools/python/lib/python2.7/site-packages/checkm/genomicSignatures.py", line 185, in calculate
p.terminate()
File "/global/common/genepool/usg/languages/python/2.7.10/lib/python2.7/multiprocessing/process.py", line 137, in terminate
self._popen.terminate()
AttributeError: 'NoneType' object has no attribute 'terminate'

[donovan-h-parks]

If you can send me a simplified data set that causes the error that would be great. I've yet to encounter this issue on my end.

[jeffkimbrel]

I am also having this problem after not having it previously. I am using the bioconda version of checkm inside of a python 3.6 environment initiated with this:

# set up channels
conda config --add channels defaults
conda config --add channels conda-forge
conda config --add channels bioconda

# install
conda install checkm-genome
conda install hmmer
conda install prodigal
conda install pplacer

The checkM pipeline previously worked with this command:

checkm lineage_wf -t 8 -x fasta bins checkM_out

I added the --pplacer_threads 32 command, and bumped up to -t 32, and checkM failed at analyze saying it couldn't find hmmer commands.

[Error] Make sure HMMER executables (e.g., hmmsearch, hmmfetch) are on your system path.

I tried reducing the number of threads, but only worked after completely removing the pplacer commands and dropping the threads to 8.

Failed - checkm lineage_wf -t 32 --pplacer_threads 32 -x fasta bins checkM_out
Failed - checkm lineage_wf -t 16 --pplacer_threads 16 -x fasta bins checkM_out
Failed - checkm lineage_wf -t 8 --pplacer_threads 8 -x fasta bins checkM_out
Failed - checkm lineage_wf -t 32 -x fasta bins checkM_out
Success - checkm lineage_wf -t 8 -x fasta bins checkM_out

So it looks like there may be two different issues (number of threads and including pplacer_threads). The bioconda hmmer says it has both 2.3.2 & 3.1b2 versions with one install, so perhaps at some point 2.3.2 is being attempted?

[donovan-h-parks]

Is it failing with the "HMMER executable" error in all cases? I suspect not. Increasing the number of threads increases the amount of memory required by CheckM. I suspect that you are running out of memory when using multiple threads. How much memory is on your machine?