Differences in "Existing_variation" field when using indexed/unindexed cache

[smijalko]

Hello,

I see a difference in some variants when I annotate the same VCF with indexed and unindexed cache with the same parameter setup.

Here is the command line used for running VEP:

tar xfz homo_sapiens_vep_95_GRCh38.tar.gz -C /opt/ensembl-vep/ && perl -I /root/.vep/Plugins/ /opt/ensembl-vep/vep --offline --dir /opt/ensembl-vep --buffer_size 500000 --cache --force_overwrite --fork 4 --input_file Strelka2_somatic.snvs.vcf.gz --no_progress --plugin Condel,/opt/ensembl-vep/plugin-files/Condel/config,b --output_file Strelka2_VEP.txt --species homo_sapiens --no_stats --polyphen b --sift b --no_escape --numbers --canonical --ccds --protein --symbol --af_1kg --af_esp --af_gnomad --check_existing --vcf

Examples of a difference where one of the caches gives more/less ids:

---

chr1 235112115 . G GA Existing_variation = rs778909032
chr1 235112115 . G GA Existing_variation = rs778909032&TMP_ESP_1_235275431_235275431

chr16 1065849 . C A Existing_variation = rs34613944&rs80286221
chr16 1065849 . C A Existing_variation = rs80286221

Command line used for indexing VEP cache:

tar xfz homo_sapiens_vep_95_GRCh38.tar.gz -C /opt/ensembl-vep/cache && /opt/ensembl-vep/convert_cache.pl --species all --version all --dir /opt/ensembl-vep/cache && tar czf homo_sapiens_vep_95_GRCh38.processed.tgz -C /opt/ensembl-vep/cache .

Can you please help me understand what causes those differences? Is this expected?

Best,
Sanja

[aparton]

Hi, Thanks for sending this issue on. We’re currently in the process of investigating, and it looks like there is an issue in the non-indexed cache relating to multi-allelic variants. The variants reported in the indexed cache are correct, so please continue to use that one. We don’t recommend the non-indexed cache files, particularly for human, as they can be quite slow. I’ll let you know when we find a concrete solution to the non-indexed cache issue. Kind Regards, Andrew

…

On 13 Mar 2019, at 16:26, smijalko ***@***.***> wrote: Hello, I see a difference in some variants when I annotate the same VCF with indexed and unindexed cache with the same parameter setup. Here is the command line used for running VEP: tar xfz homo_sapiens_vep_95_GRCh38.tar.gz -C /opt/ensembl-vep/ && perl -I /root/.vep/Plugins/ /opt/ensembl-vep/vep --offline --dir /opt/ensembl-vep --buffer_size 500000 --cache --force_overwrite --fork 4 --input_file Strelka2_somatic.snvs.vcf.gz --no_progress --plugin Condel,/opt/ensembl-vep/plugin-files/Condel/config,b --output_file Strelka2_VEP.txt --species homo_sapiens --no_stats --polyphen b --sift b --no_escape --numbers --canonical --ccds --protein --symbol --af_1kg --af_esp --af_gnomad --check_existing --vcf Examples of a difference where one of the caches gives more/less ids: chr1 235112115 . G GA Existing_variation = rs778909032 chr1 235112115 . G GA Existing_variation = rs778909032&TMP_ESP_1_235275431_235275431 < chr16 1065849 . C A Existing_variation = rs34613944&rs80286221 chr16 1065849 . C A Existing_variation = rs80286221 Command line used for indexing VEP cache: tar xfz homo_sapiens_vep_95_GRCh38.tar.gz -C /opt/ensembl-vep/cache && /opt/ensembl-vep/convert_cache.pl --species all --version all --dir /opt/ensembl-vep/cache && tar czf homo_sapiens_vep_95_GRCh38.processed.tgz -C /opt/ensembl-vep/cache . Can you please help me understand what causes those differences? Is this expected? Best, Sanja — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#413>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AfvjN4Cf1C3X44SWqWqdWsqJcV9pDPpnks5vWSakgaJpZM4btqcC>.

[sigven]

Hi,

I just experienced a somewhat related issue (I believe), through the web interface. A particular variant (chr5:112841059T>A, grch38) yields both dbSNP annotation (rs459552) and gnomAD frequencies through the web interface. Submitting the same variant with grch37 coordinates (chr5:112176756T>A) yields no gnomAD frequences and no dbSNP reference.

Performing annotation of the same variant with my local v96 installation (I use the pre-indexed cache) gives similar results for grch37, i.e. no gnomAD frequencies and no dbSNP reference. Having the --failed 1 option turned on did not resolve the issue.

Any clues as to why this variant do not report the same annotations for grch37 and grch38?

regards,
Sigve

[aparton]

Hi @sigven,

I've taken a look at this issue this morning and it looks like there's a bug at our end where variants on chromosome 5 are missing for GRCh37.

Thanks for reporting this to us, and I'll let you know when there's a fix in place.

Kind Regards,
Andrew

[reWfLoiNDERnwARy]

D as acad cu Enviado desde mi Sony Xperia™ smartphone de Telcel

…

---- Andrew Parton wrote ---- Hi @sigven<https://github.com/sigven>, I've taken a look at this issue this morning and it looks like there's a bug at our end where variants on chromosome 5 are missing for GRCh37. Thanks for reporting this to us, and I'll let you know when there's a fix in place. Kind Regards, Andrew — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#413?email_source=notifications&email_token=AKDR3YMLKQF43QDWFJNHH7TPWPDOJA5CNFSM4G5WU4BKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV3OAKI#issuecomment-494329897>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AKDR3YKQUMRBXKJ7XSS32G3PWPDOJANCNFSM4G5WU4BA>.

[aparton]

Hi,

Just to provide an update to these issues, we're still investigating the initial cause of the differences between the indexed/non-indexed cache files.

The issue with missing variants on chromosome 5 on GRCh37 has now been resolved.

Kind Regards,
Andrew

[sigven]

Thanks for that. I do however notice that the pre-indexed for GRCh37 has increased significantly in size (almost 20Gb), significantly larger than GRCh38. I cannot seem to remember that it was like this in previous releases? Any comments to this?

regards,
Sigve

[aparton]

Hi,

This is due to the nature of the fix that we applied, to get a fix out sooner rather than later then we converted one of our non-indexed cache files, rather than generating an entirely new one. This has left additional non-indexed components in there, which will not have an effect on your output data.

The GRCh37 cache file will be restored to its original size with release 97, currently scheduled for release in early July.

Thanks,
Andrew

[sigven]

Thanks for the clarification. Will wait for the 97 release :-)
best,
Sigve

[at7]

Dear @sigven,
I would like to let you know that we have released Ensembl 97. Please start using the new cache files and let us in case you notice any problems.
Best wishes,
Anja

[sigven]

Dear @at7,
Thanks for the friendly reminder:-) I have already started looking at them, will let you know if I run into any issues.

best,
Sigve