liftovervcf error for large vcf files with genotypes

[chenwenan]

Bug Report

Affected tool(s)

picard

Affected version(s)

2.17.0

Description

When running picard LiftoverVcf on a vcf file of 45G with genotypes, I have the following error:

Runtime.totalMemory()=21361459200
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
at java.lang.StringBuilder.append(StringBuilder.java:136)
at htsjdk.tribble.util.ParsingUtils.split(ParsingUtils.java:276)
at htsjdk.variant.vcf.AbstractVCFCodec.decodeLine(AbstractVCFCodec.java:275)
at htsjdk.variant.vcf.AbstractVCFCodec.decode(AbstractVCFCodec.java:262)
at htsjdk.variant.vcf.AbstractVCFCodec.decode(AbstractVCFCodec.java:64)
at htsjdk.tribble.TabixFeatureReader$FeatureIterator.readNextRecord(TabixFeatureReader.java:180)
at htsjdk.tribble.TabixFeatureReader$FeatureIterator.next(TabixFeatureReader.java:215)
at htsjdk.tribble.TabixFeatureReader$FeatureIterator.next(TabixFeatureReader.java:155)
at picard.vcf.LiftoverVcf.doWork(LiftoverVcf.java:271)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:268)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:98)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:108)

Steps to reproduce

Expected behavior

I guess the large vcf files could be handled by storing only part of the variants in memory

Actual behavior

out of memory error

[yfarjoun]

I think that the problem here is using the wrong value for MAX_RECORDS_IN_RAM....am I correct?

[chenwenan]

I think I simply used the default without specifying this parameter. Will this parameter help?

[yfarjoun]

yes, for VCF sorting it's best to use a smaller value. Try 10000.

[merceneryinbox]

In my case such explicit reducing not effective with large vcf...