Error: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory

[IsmailM]

Running gemBS extract (jobs = 16) on a smaller node (16 threads and 64gb ram) fails with the following error:

:
: Command extract started at 2018-10-17 13:27:31.752542
:
: Methylation Extraction...
2018-10-17 17:07:39,543 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13
2018-10-17 17:07:40,243 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13
2018-10-17 17:07:39,619 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13
2018-10-17 17:07:39,522 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13
2018-10-17 17:07:39,486 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13
2018-10-17 17:07:39,599 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13
2018-10-17 17:07:39,511 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13
2018-10-17 17:07:39,586 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13
2018-10-17 17:07:39,570 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13
2018-10-17 17:07:40,155 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13
2018-10-17 17:07:40,276 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13
2018-10-17 17:07:41,024 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory
2018-10-17 17:07:41,045 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory
2018-10-17 17:07:41,045 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory
2018-10-17 17:07:41,045 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory
2018-10-17 17:07:41,088 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory
2018-10-17 17:07:41,155 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory
2018-10-17 17:07:41,155 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory
2018-10-17 17:07:41,156 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory
2018-10-17 17:07:41,227 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory
2018-10-17 17:07:41,242 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory
2018-10-17 17:07:41,242 ERROR: Could not write 4096 bytes: Error 1
2018-10-17 17:07:41,262 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory
2018-10-17 17:07:41,262 ERROR: Could not write 4096 bytes: Error 1
2018-10-17 17:07:41,272 ERROR: Could not write 4096 bytes: Error 1
2018-10-17 17:07:41,272 ERROR: Could not write 4096 bytes: Error 1
2018-10-17 17:07:41,272 ERROR: Could not write 4096 bytes: Error 1
2018-10-17 17:07:41,272 ERROR: Could not write 4096 bytes: Error 1
Exception in thread Thread-7:
Traceback (most recent call last):
ValueError: Error while extracting SNP calls.

Exception in thread Thread-6:
Traceback (most recent call last):
ValueError: Error while extracting SNP calls.

Exception in thread Thread-4:
Traceback (most recent call last):
ValueError: Error while extracting SNP calls.

Exception in thread Thread-2:
Traceback (most recent call last):
ValueError: Error while extracting SNP calls.

Exception in thread Thread-3:
Traceback (most recent call last):
ValueError: Error while extracting SNP calls.

2018-10-17 17:07:41,286 ERROR: Could not write 4096 bytes: Error 1
Exception in thread Thread-5:
Traceback (most recent call last):
ValueError: Error while extracting SNP calls.

2018-10-17 17:07:41,302 ERROR: Could not write 4096 bytes: Error 1
Exception in thread Thread-11:
Traceback (most recent call last):
ValueError: Error while extracting SNP calls.

Exception in thread Thread-1:
Traceback (most recent call last):
ValueError: Error while extracting SNP calls.

2018-10-17 17:07:41,312 ERROR: Could not write 4096 bytes: Error 1
Exception in thread Thread-9:
Traceback (most recent call last):
ValueError: Error while extracting SNP calls.

2018-10-17 17:07:41,313 ERROR: Could not write 4096 bytes: Error 1
2018-10-17 17:07:41,372 ERROR: Could not write 4096 bytes: Error 1
Exception in thread Thread-10:
Traceback (most recent call last):
ValueError: Error while extracting SNP calls.

Exception in thread Thread-8:
Traceback (most recent call last):
ValueError: Error while extracting SNP calls.

The relevant lines from the config:

[extract]
# there is a jobs arg but no threads arg
jobs=16
# strand_specific = True
phred_threshold = 10
make_cpg = True
make_non_cpg = True
make_bedmethyl = True
# make_bigwig = True
make_snps = True

[heathsc]

I would suggest running with less jobs given it it looks like you are running out of memory. Simon

…

On 19 Oct 2018, at 11:43, Ismail Moghul ***@***.***> wrote: Running gemBS extract (jobs = 16) on a smaller node (16 threads and 64gb ram) fails with the following error: : : Command extract started at 2018-10-17 13:27:31.752542 : : Methylation Extraction... 2018-10-17 17:07:39,543 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13 2018-10-17 17:07:40,243 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13 2018-10-17 17:07:39,619 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13 2018-10-17 17:07:39,522 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13 2018-10-17 17:07:39,486 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13 2018-10-17 17:07:39,599 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13 2018-10-17 17:07:39,511 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13 2018-10-17 17:07:39,586 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13 2018-10-17 17:07:39,570 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13 2018-10-17 17:07:40,155 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13 2018-10-17 17:07:40,276 ERROR: Process '/usr/local/lib/python3.6/dist-packages/gemBS/bin/bcftools' finished with -13 2018-10-17 17:07:41,024 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory 2018-10-17 17:07:41,045 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory 2018-10-17 17:07:41,045 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory 2018-10-17 17:07:41,045 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory 2018-10-17 17:07:41,088 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory 2018-10-17 17:07:41,155 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory 2018-10-17 17:07:41,155 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory 2018-10-17 17:07:41,156 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory 2018-10-17 17:07:41,227 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory 2018-10-17 17:07:41,242 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory 2018-10-17 17:07:41,242 ERROR: Could not write 4096 bytes: Error 1 2018-10-17 17:07:41,262 ERROR: [E::bgzf_compress] Call to deflateInit2 failed: insufficient memory 2018-10-17 17:07:41,262 ERROR: Could not write 4096 bytes: Error 1 2018-10-17 17:07:41,272 ERROR: Could not write 4096 bytes: Error 1 2018-10-17 17:07:41,272 ERROR: Could not write 4096 bytes: Error 1 2018-10-17 17:07:41,272 ERROR: Could not write 4096 bytes: Error 1 2018-10-17 17:07:41,272 ERROR: Could not write 4096 bytes: Error 1 Exception in thread Thread-7: Traceback (most recent call last): ValueError: Error while extracting SNP calls. Exception in thread Thread-6: Traceback (most recent call last): ValueError: Error while extracting SNP calls. Exception in thread Thread-4: Traceback (most recent call last): ValueError: Error while extracting SNP calls. Exception in thread Thread-2: Traceback (most recent call last): ValueError: Error while extracting SNP calls. Exception in thread Thread-3: Traceback (most recent call last): ValueError: Error while extracting SNP calls. 2018-10-17 17:07:41,286 ERROR: Could not write 4096 bytes: Error 1 Exception in thread Thread-5: Traceback (most recent call last): ValueError: Error while extracting SNP calls. 2018-10-17 17:07:41,302 ERROR: Could not write 4096 bytes: Error 1 Exception in thread Thread-11: Traceback (most recent call last): ValueError: Error while extracting SNP calls. Exception in thread Thread-1: Traceback (most recent call last): ValueError: Error while extracting SNP calls. 2018-10-17 17:07:41,312 ERROR: Could not write 4096 bytes: Error 1 Exception in thread Thread-9: Traceback (most recent call last): ValueError: Error while extracting SNP calls. 2018-10-17 17:07:41,313 ERROR: Could not write 4096 bytes: Error 1 2018-10-17 17:07:41,372 ERROR: Could not write 4096 bytes: Error 1 Exception in thread Thread-10: Traceback (most recent call last): ValueError: Error while extracting SNP calls. Exception in thread Thread-8: Traceback (most recent call last): ValueError: Error while extracting SNP calls. The relevant lines from the config: [extract] # there is a jobs arg but no threads arg jobs=16 # strand_specific = True phred_threshold = 10 make_cpg = True make_non_cpg = True make_bedmethyl = True # make_bigwig = True make_snps = True — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#38>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADHPdx0vb14RrXC0g0vutS4drqsl3PRMks5umZ6qgaJpZM4XwBVd>.

[IsmailM]

Thanks, is there a rule of thumb on deciding how much memory to request before submitting a job? The bam files were roughly 11 x 70gb and the bcf files were roughly 11 x 33gb

[jingydz]

[E::bgzf_compress] Call to deflateInit2 failed: insufficient memory

I encountered this issue as well; the error occurred when I attempted to extract a subset of samples from a VCF file using bcftools. It seems to be due to insufficient memory. When I switched to a node with more memory, the problem was resolved.