Support GRCh38

[schelhorn]

I was wondering if there are any plans to support GRCh38 as an additional assembly. The alternate loci and decoy sequences are likely to improve both variant calling and expression studies. I suppose that the availability of genomes and annotations are not much of a problem anymore, but not all tools may be compatible. Could someone detail the major roadblocks concerning this issue? Thanks.

[chapmanb]

Sven-Eric;
We'd love to support GRCh38 and I agree that it's a much improved build. The only major blocker is that we don't currently have any reference standards to validate against so any update would be blind to the performance of the callers or our implementation. The Genome in a Bottle consortium is currently working on this so hopefully this is only a short term issue.

Beyond that, it's a matter of putting together the time and resources to work on this. The major steps we need to do are:

• Add data preparation for the reference build and associated files (dbSNP and friends). We've been using the Broad bundles for other human builds but they haven't yet moved to GRCh38.
• Add data preparation for GRCh38 reference genomes for RNA-seq.
• Update bwa-mem code to be aware and take into account alternative locus mapping (https://github.com/lh3/bwa/blob/master/README-alt.md).
• Testing to ensure callers work cleanly with new build.
• Updating downstream tools we rely on, like GEMINI, to contain annotations for GRCh38.

We're hopeful to get a project that needs/asks for GRCh38 we could use to fund the work on this, or happy to work along slowly as we have time to add it in. Thanks for the discussion.

[schelhorn]

Great, thanks for the comprehensive answer. It's good to be conservative about references and I for one am happy to wait for validation standards and GEMINI support if helps us to secure the validity of the results. Please feel free to close this issue. Perhaps it could make sense to copy the answer to the documentation in case anyone else is looking for it?

[chapmanb]

Sven-Eric, definitely want to keep this moving even if slowly. To get the process going here are some notes on GRCh38 resources. We can continue to collect them here and merge into the automated data installation. ccing @arq5x and @brentp in case you've started working on resource collection/installation for GEMINI already as I'd love to coordinate with you:

• FASTA file with decoy, ready for bwa-mem, including bwa and bowtie indices:
  ftp://ftp.ncbi.nlm.nih.gov/genbank/genomes/Eukaryotes/vertebrates_mammals/Homo_sapiens/GRCh38/seqs_for_alignment_pipelines/

  With alternatives: GCA_000001405.15_GRCh38_full_plus_hs38d1_analysis_set.fna
  No alternatives alleles: GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.gz

• dbSNP for GRCh38:
  http://www.ncbi.nlm.nih.gov/variation/docs/human_variation_vcf/

  Full: 00-All.vcf.gz

• Annotations from Ensembl:
  ftp://ftp.ensembl.org/pub/release-79/gtf/homo_sapiens/

  release79: Homo_sapiens.GRCh38.79.gtf.gz
  Needs to be converted to use chr style prefixes instead of bare to maintain
  consistency with NCBI and UCSC.

• COSMIC (licensing restrictions with v72):
  sftp://sftp-cancer.sanger.ac.uk/files/grch38/cosmic/v72/VCF/CosmicCodingMuts.vcf.gz
  sftp://sftp-cancer.sanger.ac.uk/files/grch38/cosmic/v72/VCF/CosmicNonCodingVariants.vcf.gz

• ClinVar:
  ftp://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/

[arq5x]

We haven’t yet started coordinating annotations for GRCh38, but this is a very useful collection. My big concern is how to port datasets such as ESP and ExAC, as they will almost certainly not be recalled on GRCh38!

Aaron

On Apr 14, 2015, at 2:12 PM, Brad Chapman notifications@github.com wrote:

Sven-Eric, definitely want to keep this moving even if slowly. To get the process going here are some notes on GRCh38 resources. We can continue to collect them here and merge into the automated data installation. ccing @arq5x and @brentp in case you've started working on resource collection/installation for GEMINI already as I'd love to coordinate with you:

FASTA file with decoy, ready for bwa-mem, including bwa and bowtie indices:
ftp://ftp.ncbi.nlm.nih.gov/genbank/genomes/Eukaryotes/vertebrates_mammals/Homo_sapiens/GRCh38/seqs_for_alignment_pipelines/

With alternatives: GCA_000001405.15_GRCh38_full_plus_hs38d1_analysis_set.fna
No alternatives alleles: GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.gz

dbSNP for GRCh38:
http://www.ncbi.nlm.nih.gov/variation/docs/human_variation_vcf/

Full: 00-All.vcf.gz

Annotations from Ensembl:
ftp://ftp.ensembl.org/pub/release-79/gtf/homo_sapiens/

release79: Homo_sapiens.GRCh38.79.gtf.gz
Needs to be converted to use chr style prefixes instead of bare to maintain
consistency with NCBI and UCSC.

COSMIC (licensing restrictions with v72):
sftp://sftp-cancer.sanger.ac.uk/files/grch38/cosmic/v72/VCF/CosmicCodingMuts.vcf.gz
sftp://sftp-cancer.sanger.ac.uk/files/grch38/cosmic/v72/VCF/CosmicNonCodingVariants.vcf.gz

ClinVar:
ftp://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/

—
Reply to this email directly or view it on GitHub.

[chapmanb]

Thanks Aaron -- we're going to have to hold our noses and get familiar with liftOver and friends, at least in the medium term. It looks like there are 3 different tools:

• Ensembl: http://useast.ensembl.org/Homo_sapiens/Tools/AssemblyConverter?db=core
• UCSC: https://genome.ucsc.edu/cgi-bin/hgLiftOver
• NCBI: http://www.ncbi.nlm.nih.gov/genome/tools/remap

and some relevant discussion:

https://www.biostars.org/p/65558/
https://www.biostars.org/p/113184/

Does anyone have good/bad experience with any of these?

[chapmanb]

From @druvus -- notes from SciLife's work on building a GATK bundle for 38:

https://wabi-wiki.scilifelab.se/display/SHGATG/gatk+bundle+in+hg38

[mw55309]

Hi Guys

Bringing in Alison Meynert (@ameynert) who is one of our medical genomics experts in Edinburgh. She has been working on lift over of GRCh37 GATK resources to GRCh38.

Cheers
Mick

[ameynert]

I've been running CrossMap, and see similar results to what the SciLife post reported. Nearly all sites can be mapped to GRCh38 for all the GATK resources with the Ensembl chain file:

0.9983 1000G_omni2.5.b37.vcf
0.9974 1000G_phase1.indels.b37.vcf
0.9986 1000G_phase1.snps.high_confidence.b37.vcf
0.9961 dbsnp142.b37.vcf
0.9986 hapmap_3.3.b37.vcf
0.9976 Mills_and_1000G_gold_standard.indels.b37.vcf

I'll take a look at the breakdown of missing vs. incorrectly mapped for the dbSNP benchmark, and run dbSNP138 as well as 142.

[arq5x]

Very interesting, thanks! I expect that this approach will work equally as well for ESP, Clinvar, and the ExAC dataset. However, I wonder how appropriate lift over is for annotations such as CADD and fitCons where local genomic context factors into the scores assigned. In other words, the new location is likely not in doubt, but the question is whether the score calculated for b37 is appropriate for b38. I suspect tht there are many cases where this is not the case.

[chapmanb]

Alison and Mick -- thanks so much for investigating the CrossMap approach. Those numbers looks great, I'd be happy to include the GATK supporting files into bcbio so we have them available. We can always iterate and update as we continue to evaluate these.

Practically, I added in downloads of hg38/GRCh38 (without alternative sequences) to bcbio. If you update to the latest version of the development code you can do:

bcbio_nextgen.py upgrade -u development
bcbio_nextgen.py upgrade --data --genomes hg38-noalt

and it will install the reference genome, bwa/bowtie2 indices, and dbSNP 142. The references and indices all come from NCBI:

ftp://ftp.ncbi.nlm.nih.gov/genbank/genomes/Eukaryotes/vertebrates_mammals/Homo_sapiens/GRCh38/seqs_for_alignment_pipelines/README_ANALYSIS_SETS

I haven't done any runs with the hg38 genome but having the basic data installation in place is step one.

Aaron, on the logistics side I used your GGD approach to manage the installation. The separation between the download/prep code and the machinery to actually do the installations is nice -- it makes the data provenance clear. I needed to stray from your initial GGD implementation of streaming the results of one download to one file since these preps involved picking and prepping multiple files from the initial tarballs. I implemented a take on GGD which uses a temporary directory, runs all of the file prep commands as a bash script, then pulls out the final files into the output directory when updated:

https://github.com/chapmanb/cloudbiolinux/tree/master/ggd-recipes
https://github.com/chapmanb/cloudbiolinux/blob/master/cloudbio/biodata/ggd.py

I'd love to try and coordinate this with what you want to do for GEMINI so we can have a single approach to get one set of downloaded data.

Thanks all for the discussion and ideas.

[druvus]

Hi,
just want to bring in Pall Olason (@pallolason) who did the liftover work at SciLifeLabs into the discussion.

cheers,
Andreas

[ameynert]

The GATK bundle file of dbSNP138 excluding 1000 Genomes samples should also get lifted over for use with VQSR. There are issues with tranche calculations caused by hard-coded expectations of pre-1KG data. See the last comment on the GATK forum here: http://gatkforums.broadinstitute.org/discussion/5221/tranches-plot-issue

The presenters at this week's GATK roadshow said they were hoping to have this fixed in release 3.4.

[pallolason]

Thanks for the heads up @druvus.
@ameynert , did the Ensembl chain files work for you out of the box with CrossMap? When I use them to map vcf files, I have to edit the "target" genome chromosome names to "chrN" style. Otherwise CrossMap will not fetch the REF allele from the fasta file. I reproduced that with the CrossMap tool on Ensembls assembly converter page: http://www.ensembl.org/Homo_sapiens/Tools/AssemblyConverter?db=core

[ameynert]

@pallolason The chain files worked out of the box for coordinate mapping only, but not for the reference bases, as you found. I'm trying the UCSC hg19 to hg38 chain files to see how those compare to hacking the Ensembl chain files to use 'chrN' style coordinates. It's strange that CrossMap has this limitation since the VCF files I'm converting are all in 'N' chromosome style.

[chapmanb]

Alison -- for GATK VQSR plotting compatible dbSNPs, we could subset the existing NCBI GRCh38 dbSNP VCF. It has dbSNPBuildID for the introduced version, plus KGPhase1 and KGPhase3 to indicate 1000genomes versions. That'll help us create a subset file directly from the already lifted over version instead of doing it twice. Since it's just a plotting issue it might be worth waiting until 3.4 and seeing what the GATK folks recommend before doing this.

When you think the lifted over supplemental files are ready I'll be happy to include them in the bcbio distribution. Just let me know.

For the full GRCh38 build plus alts, we'll plan to use the distribution from 1000 genomes:

ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/GRCh38_reference_genome/

This is the NCBI full build plus additional HLAs from @lh3 which will hopefully get us HLA typing:

https://github.com/lh3/bwa/blob/master/README-alt.md#hla-typing