Add REVEL Score

[snesic]

Feature Requirements:

Add REVEL score to VarFish so that the variants can be sorted and filtered by this score.

Use cases:

1. Display REVEL score in "extra annotations" box in variant detail fold-out.
2. Display REVEL score in the filtration results table in each row for each missense variant.
   The display should be in the same way as population frequency.
3. Allow filtration by REVEL score threshold, i.e., we need to add it to the filter form.
   • A more detailed discussion yielded that this is not strictly required and we can just sort by the REVEL score and just look at the top N variants by descending REVEL score or stop considering variants with REVEL score below X.
4. Download TSV/Excel file that contains REVEL score, just as they have population frequencies.

Design Overview

Different strategies to implement this are discussed here. We decided to follow the option where the REVEL score is added to the extra annotations table. We further want to allow annotation of arbitrary values from the extra annotations in the result table (and downloads) and allow filtration by these values.

Steps

1. Extra Annotations Data Update Get REVEL score into extra annotations (TSV and table).
2. Extra Annotations Into Results Get extra annotation values form extra annotations table into VarFish results.
   At this point, the extra annotations such as the REVEL score is visible to users and users can already sort in the result. We also need to ensure that the export works here.
   Care needs to be taken that the results table does not explode in terms of number of columns. Users should be able to show/hide the extra annotation columns similar to how it's done for columns such as "distance to splice site."
3. Extra Annotation Tresholds Extend the filter form (and supporting) code such that it is aware of the extra annotations and thresholds can be set.
   • See comment above, no need to implement this at the moment.

Depencies

• (1.) indepent
• (3.) -> (2.)

Notes

• The list of extra annotations is given by table ExtraAnnotationField.
• Always annotate all extra annotations scores, all are hidden by default and the "Columns" Multi-Select allows to show them.
  

Design Details

1. Extra Annotations Data Update

• Develop a small utility to take the "vanilla" (originally released) ExtraAnnos.tsv and ExtraAnnoFields.tsv files and augment them with additional scores. The "additional scores" could/should come from a TSV file with (genome-build, chromosome, 1-based position, reference, alternative, ...) columns.
• This will allow users to create their own augmented extra annotation tables beyond the REVEL score.
• In the future, the varfish-db-downloader Snakefiles could incorporate the most "popular" scores but for now we want do not want to add every possible score in there.
• The annotation utility itself can live in varfish-db-downloader repository.

INPUT

# cat score-to-augment.tsv
chr1 1000000 G A 37.5

# cat ExtraAnno.tsv
[...]
GRCh37  1       1000000 1000000 592     G       A       [27.33, 8.54, 0.72, 0.48, 8.2, 2.7, 5.02, 2.91, 52.21, 0, 0, 0]
[...]

# cat ExtraAnnoField.tsv 
field   label
1       EncodeH3K27me3-sum
2       EncodeH3K27me3-max
3       EncodeH3K36me3-sum
4       EncodeH3K36me3-max
5       EncodeH3K79me2-sum
6       EncodeH3K79me2-max
7       EncodeH4K20me1-sum
8       EncodeH4K20me1-max
9       EncodeH2AFZ-sum
10      MMSp_acceptorIntron
11      MMSp_acceptor
12      MMSp_donor

USAGE

augment-extra-annotations.py \
    --input-annos=ExtraAnno.tsv \
    --input-annos-fields=ExtraAnnoFields.tsv \
    --input-score=score-to-augment.tsv \
    --genomebuild=GRCh37 \
    --field-name=REVEL_score \
    --output-annos=ExtraAnnos_with_REVEL.tsv \
    --output-annos-fields=ExtraAnnosFields_with_REVEL.tsv

RESULT

# cat ExtraAnno.tsv
GRCh37  1       1000000 1000000 592     G       A       [27.33, 8.54, 0.72, 0.48, 8.2, 2.7, 5.02, 2.91, 52.21, 0, 0, 0, 37.5]

# cat ExtraAnnoField.tsv 
field   label
1       EncodeH3K27me3-sum
2       EncodeH3K27me3-max
3       EncodeH3K36me3-sum
4       EncodeH3K36me3-max
5       EncodeH3K79me2-sum
6       EncodeH3K79me2-max
7       EncodeH4K20me1-sum
8       EncodeH4K20me1-max
9       EncodeH2AFZ-sum
10      MMSp_acceptorIntron
11      MMSp_acceptor
12      MMSp_donor
13      REVEL_score

Affected Components

• Additional script in varfish-db-downloader with documentation.
• No change to varfish-db-downloader workflows or existing files.

2. Extra Annotations Into Results

(a) augmenting the query infrastructure

• Augment variants.query such that the information from the ExtraAnno model is available in the results
• Use ExtendQueryPartsCommentsJoin as a model (copy-paste-and-adjust) => ExtendQueryPartsExtraAnnoJoin
• Develop in test-driven way, start with writing tests.

(b) adjust result table templates

• paths: variants/templates/variants/filter_result/{header,row}.html
• write all extra annotation fields in header and each row
• use JQuery/CSS trick for hiding the columns that should not be displayed (e.g., look how it's done for detail-frequencies-thousand-genomes)
• the current selects for constraints and columns are in table.html, look for string result-columns-selector
• of course, the list of extra annotation fields must be given into the templates in variants.views

(c) for TSV/Excel downloads

• module variants.file_exports
• when (a) has been implemented, the data is available in the code that generates the download data
• augment CaseExporterBase._yield_columns and CaseExporterBase._yield_small_vars to return the extra annotations in an appropriate way (break apart the JSON list that will be in the output)
• probably it makes sense to start out with tests first

(d) adjust documentation

• describe the new feature in the documentation and also the release notes

Affected Components

• variants.queries, variants.file_exports, variants.views, possibly more modules in variants
• extend templates in variants/templates
• of course, tests as appropriate

3. Extra Annotation Thresholds

• most probably not required as results can be sorted by score
• decision: postpone for now

[holtgrewe]

As discussed in Teams,

We should change the Requirements as follows.

### Feature Requirements
1. Display REVEL score in "extra annotations" box in variant detail fold-out.
2. Display REVEL score in the filtration results table in each row for each missense variant.
   The display should be in the same way as population frequency.
3. Allow filtration by REVEL score threshold, i.e., we need to add it to the filter form.
4. Download TSV/Excel file that contains REVEL score, just as they have population frequencies.

As for the change/design proposal please update this section towards what we discussed, i.e., no full inclusion in varfish db downloader but Snakefiles, more as an add-on tool to annotate extra annotations based on a chr/pos/ref/alt TSV. Annotating extra annotations with the REVEL score would be special case of such a general purpose tool. Please also propose where to change varfish-server with regards to the requirements 1-4. This will most probably be starting points for an design/architecture change document but please provide some structure and start looking at the source code to identify suitable parts in the VarFish Server views/queries/etc.

IOW: Please rework your requirements. To answer your question from Teams, we will not have the REVEL score in the mainline of varfish-db-downloader. Rather, we should (1) create a tool to annotate extra annos TSV file with further scores from TSV/tabix files such as downloaded REVEL score and (2) provide users with instrutions on how to extend their extra annos TSV score with REVEL scores for example.

We further need a more refined plan of the required changes in varfish-server. I don't see a need for a new table and query engine performance is not so critical, we will need to augment then design. Please start by studying the query engine source code.

[stolpeo]

@holtgrewe @snesic Specifications are looking good to me. Just some comments:

Tool: I would emphasize, for the tool interface please use argparse (code can be copied from other tools in the DB downloader, e.g. https://github.com/bihealth/varfish-db-downloader/blob/master/tools/knowngeneaa.py ). Also, please locate the tool in the tools subfolder.

Query: Getting into the query structure can be daunting at first, but the actual change is not too complicated. I wondered if in the long term there could be a generic or automatic solution to add extra annotation fields to the query.

Tests: For the planning, setting up tests will probably take at least as long as the actual implementation, if not longer.