Vcf export #241

ch-kr · 2020-07-22T14:32:07Z

Added constants and some functions that are used during MT/HT>VCF export

…ters_sanity_check_expr to vcf.py from gnomad_qc. Also added INFO_DICT, FORMAT_DICT, and make_vcf_filter_dict from ukb repo

…_dict to update INFO_DICT with AS fields and their values

…IELDS, REGION_TYPE_FIELDS, ALLELE_TYPE_FIELDS

… from docstrings

jkgoodrich

Thank you for all your hard work on this KC! I have some suggested changes and comments.

gnomad/utils/vcf.py

jkgoodrich · 2020-07-24T23:36:36Z

gnomad/utils/vcf.py

+    return header_hist_dict
+
+
+def sample_sum_check(


This is something we want to check on the internal HT sanity checks right? I think it should also be moved somewhere else like the other function I mentioned above, but again not positive where to put it

Agreed, this could go into an assessment/sanity_checks.py type file

Yeah, I think this should move to keep this script VCF focused and assessment/sanity_checks.py sounds good to me for this and the one above it

gnomad/utils/vcf.py

…ts/their descriptions

gtiao

The examples in the docstrings are great -- having examples is clarifying in a way that describing things often is not!

gnomad/utils/vcf.py

gtiao · 2020-07-30T20:13:50Z

gnomad/utils/vcf.py

+    ]
+
+    if faf:
+        female_metrics.extend([x for x in female_metrics if "faf" in x])


I'm confused by this line: it looks like you're extending the existing list female_metrics by faf elements that are already in the list

faf elements aren't in the list -- female_metrics first pulls all annotations that contain _female and then corrects that list to keep only elements with both _female and AC, AN, or nhomalt. the faf annotations (faf95_female) don't contain AC, AN, or nhomalt, so they aren't kept in that step.

that said, why do we set AC, AN, and nhomalt to 0 but leave AF? I'm going to change this to just be female_metrics = [x for x in metrics if "_female" in x] if that works with you and @jkgoodrich

hmm, interesting that AF is not set to 0. I will defer to @gtiao for this one, there may have been motivation for this that I am not aware of.

…ts/their descriptions

jkgoodrich

A few more smaller changes requested, but it's looking really good. Thank you @ch-kr!

jkgoodrich · 2020-07-31T17:54:37Z

gnomad/utils/vcf.py

+    return header_hist_dict
+
+
+def sample_sum_check(


Yeah, I think this should move to keep this script VCF focused and assessment/sanity_checks.py sounds good to me for this and the one above it

gnomad/utils/vcf.py

jkgoodrich · 2020-07-31T20:24:58Z

gnomad/utils/vcf.py

+            ht_orig.select(*display_fields).show()
+
+
+def make_filters_sanity_check_expr(ht: hl.Table) -> Dict[str, hl.expr.Expression]:


agree, let's make an assessment/sanity_checks.py

gnomad/utils/vcf.py

jkgoodrich · 2020-07-31T21:33:01Z

gnomad/utils/vcf.py

+    ]
+
+    if faf:
+        female_metrics.extend([x for x in female_metrics if "faf" in x])


hmm, interesting that AF is not set to 0. I will defer to @gtiao for this one, there may have been motivation for this that I am not aware of.

…heck, make_filters_sanity_check_expr to assessment

jkgoodrich

Thank you for all of this @ch-kr! I only had one small comment (that has more to do with UKBB than the common code actually). Then I think there might have been a comment/s where @gtiao was tagged for opinion, but otherwise this looks good to me.

…heck_expr

gtiao

Looks great! So excited to have all this beautiful, shiny, clean code merged into the common repo!

gtiao · 2020-08-04T14:53:37Z

gnomad/utils/vcf.py

            combo_fields = combo.split("_")
+            group_dict = dict(zip(group_types, combo_fields))
+
+            for_combo = make_combo_header_text("for", group_dict, prefix)


Great idea for streamlining the code a bit!

gtiao · 2020-08-04T15:06:02Z

gnomad/utils/vcf.py

-        f"{prefix}hom": "|".join(
-            map(lambda x: f"{x:.1f}", ht.head(1).age_hist_hom.collect()[0].bin_edges)
-        ),
+        f"{prefix}{call_type}": "|".join(


Another nice consolidation of code

gtiao · 2020-08-04T15:14:45Z

gnomad/assessment/sanity_checks.py

+    Displays the number of rows in the Table that match the `cond_expr` and fail to be the desired condition (`check_description`). 
+    If the number of rows that match the `cond_expr` is 0, then the Table passes that check; otherwise, it fails.
+
+    .. note::


Thank you for adding this note -- it's probably something easily missed when setting up the generic field check, so it's great to have this additional reminder

ch-kr added 15 commits July 21, 2020 11:56

Added constants, make_label_combos, generic_field_check, and make_fil…

60b8640

…ters_sanity_check_expr to vcf.py from gnomad_qc. Also added INFO_DICT, FORMAT_DICT, and make_vcf_filter_dict from ukb repo

Added AS_FIELDS, SITE_FIELDS constants and added function add_as_info…

3decb5f

…_dict to update INFO_DICT with AS fields and their values

Updated make_as_info_dict and added more constants (RF_FIELDS, VQSR_F…

c4ad6b2

…IELDS, REGION_TYPE_FIELDS, ALLELE_TYPE_FIELDS

Added constants for entries to select during export

6c9b95f

Added make_info_dict to vcf.py

a9c0354

Removed unnecessary pop constants (can be imported from ancestry.py)

20f5d52

Added make hist bin edges expr function to vcf.py, also removed types…

2117d82

… from docstrings

Added make hist dict

78084a7

Updated changelog

5c698b7

Added SORT_ORDER and sample_sum_check to vcf.py

e30d67b

Moved make combo header text to vcf.py

706864d

Fixed imports and reformatted with black

d058743

Changed docstring for make_hist_bin_edges_expr

d1bbacd

Added set female metrics to NA

424fcc1

Updated docstring in set female metrics to na

d397b87

ch-kr requested a review from jkgoodrich July 23, 2020 21:08

jkgoodrich requested changes Jul 24, 2020

View reviewed changes

ch-kr added 10 commits July 28, 2020 12:15

Removed transmitted singleton and sibling singleton from SITE_FIELDS

ec0ddd6

Updated make_label_combos docstring and fixed values for some constan…

647d8e6

…ts/their descriptions

Updated docstring for generic_field_check

973c5ef

Updated docstring for make_filters_sanity_check_expr

ae3fcf4

Updated make_filters_sanity_check_expr

d3967aa

Updated docstring for make_combo_header_text

5313ea1

Updated docstring for make_info_dict

b556441

Updated make_combo_header_text to take dict as input

13ac89e

Updated make_info_dict to pass in sort order

1202e2e

Addressed rest of review comments

759cfac

ch-kr requested a review from jkgoodrich July 29, 2020 13:57

Add BaseQRankSum to SITE_FIELDS const

fb14af8

gtiao suggested changes Jul 30, 2020

View reviewed changes

ch-kr added 19 commits July 31, 2020 10:35

Moved make combo header text to vcf.py

00b92ec

Fixed imports and reformatted with black

6ed92b0

Changed docstring for make_hist_bin_edges_expr

191b073

Added set female metrics to NA

d8a3e7a

Updated docstring in set female metrics to na

51a5a73

Removed transmitted singleton and sibling singleton from SITE_FIELDS

6b1969d

Updated make_label_combos docstring and fixed values for some constan…

2af3587

…ts/their descriptions

Updated docstring for generic_field_check

ca57708

Updated docstring for make_filters_sanity_check_expr

4428b0c

Updated make_filters_sanity_check_expr

75e997d

Updated docstring for make_combo_header_text

fdfed1a

Updated docstring for make_info_dict

2c47c63

Updated make_combo_header_text to take dict as input

41fb66d

Updated make_info_dict to pass in sort order

2ddd8f1

Addressed rest of review comments

3a80a41

Add BaseQRankSum to SITE_FIELDS const

1e48d12

Addressed review comments

ff61379

Updated docstring for sample_sum_check

ca49ac5

Rebasing branch

5a27577

jkgoodrich requested changes Jul 31, 2020

View reviewed changes

Updated some docstrings addressing review comments

261c38d

jkgoodrich mentioned this pull request Aug 3, 2020

Vcf release broadinstitute/ukbb_qc#118

Merged

ch-kr added 3 commits August 3, 2020 13:25

Created assessment folder and moved sample_sum_check, generic_field_c…

d176f68

…heck, make_filters_sanity_check_expr to assessment

Forgot to commit changes in vcf.py when moving to assessment

1640b3f

Updated generic field check docstring

6e67800

jkgoodrich approved these changes Aug 3, 2020

View reviewed changes

Added option to check for additional filters to make_filters_sanity_c…

f021255

…heck_expr

gtiao approved these changes Aug 4, 2020

View reviewed changes

gtiao merged commit d15c4d1 into master Aug 4, 2020

gtiao deleted the vcf_export branch August 4, 2020 15:19

		ht_orig.select(*display_fields).show()


		def make_filters_sanity_check_expr(ht: hl.Table) -> Dict[str, hl.expr.Expression]:

Vcf export #241

Vcf export #241

Uh oh!

Conversation

ch-kr commented Jul 22, 2020

Uh oh!

jkgoodrich left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gtiao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jkgoodrich left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jkgoodrich left a comment

Choose a reason for hiding this comment

Uh oh!

gtiao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants