Filter to expressed variants #67

tavinathanson · 2016-06-28T18:00:31Z

In this PR:

Factored some of the df_isovar logic out of load_neoantigens into a separate _load_single_patient_isovar, to use it outside the neoantigen context.
Added _filter_variants_to_expressed to filter out any variant that isn't present in the isovar DataFrame.
Randomly made it into this branch: parameterizing variant_qc_filter.

tavinathanson · 2016-06-28T18:01:08Z

cohorts/load.py

@@ -748,13 +800,52 @@ def load_neoantigens(self, patients=None, variant_type="snv", merge_type="union"
                dfs[patient.id] = df_epitopes
        return dfs

+    def _load_single_patient_isovar(self, patient, variant_type, merge_type, epitope_lengths):


I believe the content of this is all the same; just moved.

coveralls · 2016-06-28T18:18:44Z

Coverage decreased (-0.8%) to 58.344% when pulling 394dd61 on expressed_mutations into 5b1eaf6 on master.

arahuja · 2016-06-28T19:20:24Z

cohorts/load.py

+                              ref=row["ref"],
+                              alt=row["alt"],
+                              ensembl=genome)
+            if variant in variants:


This is going to go through all of variants each time, no?

As opposed to caching the output? Yeah, I wanted to try to simplify by not caching this.

Oh, maybe you mean that I'm iterating through the VariantCollection on each check; good point.

arahuja · 2016-06-28T19:23:48Z

My thoughts on this are I'm not sure it's worth building in the isovar filtering as special case to the load_* functions and leaving it as something to do after the fact or making a filter_fn that could handle it.

tavinathanson · 2016-06-28T19:25:40Z

@arahuja curious why not?

tavinathanson · 2016-06-28T19:36:23Z

We spoke offline; @arahuja clarified that the cohorts.varcode_utils functions are more general than I realized, so I can indeed plop this logic in there.

tavinathanson · 2016-06-28T23:10:22Z

@arahuja I'm not entirely clear on the best way to do this, since my variant_expressed_filter and effect_expressed_filter need more than variant/effect and variant_metadata (the cohort, patient, variant_type, merge_type). I could put variant_expressed_filter inside a count function and make use of the closure, but that's not easily passed to load_variants.

Perhaps I should replace variant_metadata with heavier-weight object that has the VariantCollection metadata as well as cohort, patient, etc.?

arahuja · 2016-06-29T13:24:58Z

Perhaps I should replace variant_metadata with heavier-weight object that has the VariantCollection metadata as well as cohort, patient, etc.?

Perhaps? I think needing cohort would be odd, but it somewhat shows that some of the load_single functions should probably be pushed into Patient

tavinathanson · 2016-06-29T15:01:56Z

@arahuja I agree re pushing more functions to Patient; for another time, though :)

coveralls · 2016-07-09T18:02:16Z

Coverage increased (+1.0%) to 60.171% when pulling 3e4d5f8 on expressed_mutations into 5b1eaf6 on master.

coveralls · 2016-07-09T18:29:13Z

Coverage increased (+1.0%) to 60.171% when pulling 3e4d5f8 on expressed_mutations into 5b1eaf6 on master.

coveralls · 2016-07-10T14:59:24Z

Coverage decreased (-0.5%) to 60.57% when pulling 9747282 on expressed_mutations into fac5c92 on master.

coveralls · 2016-07-10T15:25:17Z

Coverage decreased (-0.5%) to 60.57% when pulling 769a0a9 on expressed_mutations into fac5c92 on master.

tavinathanson · 2016-07-10T15:39:07Z

12 days later, some updates to this PR:

Create FilterableVariant, FilterableEffect, etc. as a way to cleanly specify filter_fns as single-parameter functions. Also removes the need for neoantigen_qc_filter vs. variant_qc_filter since FilterableNeoantigen is a FilterableVariant.
Use the above to handle expression filtering instead of adding only_expressed everywhere.
Fix Spacing of new functions is off #73.
Fix VariantCollection metadata is lost after filtering #66.
Move merge_type and variant_type into Cohort so they aren't passed around to a million methods.
Add additional parameters to variant_qc_filter.

Back to you @arahuja

coveralls · 2016-07-10T17:09:34Z

Coverage decreased (-0.5%) to 60.57% when pulling a043188 on expressed_mutations into fac5c92 on master.

coveralls · 2016-07-11T19:01:44Z

Coverage decreased (-0.4%) to 60.646% when pulling 709db27 on expressed_mutations into fac5c92 on master.

coveralls · 2016-07-12T14:55:49Z

Coverage decreased (-0.4%) to 60.621% when pulling 126bda4 on expressed_mutations into fac5c92 on master.

arahuja · 2016-07-13T12:34:19Z

cohorts/load.py

@@ -215,10 +223,16 @@ def __init__(self,
                 join_how="inner",
                 check_provenance=False,
                 polyphen_dump_path=None,
-                 pageant_coverage_path=None):
+                 pageant_coverage_path=None,
+                 variant_type="snv",


any reason this is a cohort level property now?

Yeah, it became cumbersome and wordy to pass it around everywhere. Agree it's a little weird--

Yea I think I prefer the alternative, but also see the issue that you can pass one to load_variants and a different set to load_effects which could be confusing, so I'm fine with this

coveralls · 2016-07-13T12:56:38Z

Coverage decreased (-0.4%) to 60.621% when pulling 777bf4d on expressed_mutations into fac5c92 on master.

coveralls · 2016-07-13T21:08:14Z

Coverage decreased (-0.5%) to 60.714% when pulling 946eab4 on expressed_mutations into e09195b on master.

tavinathanson added the enhancement label Jun 28, 2016

tavinathanson assigned arahuja Jun 28, 2016

tavinathanson reviewed Jun 28, 2016
View reviewed changes

arahuja reviewed Jun 28, 2016
View reviewed changes

tavinathanson force-pushed the expressed_mutations branch from f9bd2b0 to ecc475c Compare July 10, 2016 14:32

tavinathanson mentioned this pull request Jul 10, 2016

Spacing of new functions is off #73

Closed

arahuja reviewed Jul 13, 2016
View reviewed changes

tavinathanson added 3 commits July 13, 2016 20:53

Filter to expressed variants

7c2d7f0

Update process_limit and comment

016a2ec

Pass variants to isovar loading function

bd93691

tavinathanson added 13 commits July 13, 2016 20:53

Pass kwargs through in expressed functions

9b1ce6c

Update filters to allow for more options

95320f7

Pass VariantCollection metadata through

305aa8c

First pass at refactoring to filter expression via filter_fn

fb43283

Remove unnecessary qc_filter functions

7178654

Fix minor bugs and tests

57a1645

Fix spacing/formatting

b8e4552

Fix linter issues

c770f9d

Formatting changes

02abe52

Fix expressed filter function

66f1c49

Remove *_qc_filter functions other than variant

bc0613a

Bug fix for Filterable objects

ee03e46

Fix weird comment quote

946eab4

tavinathanson force-pushed the expressed_mutations branch from 777bf4d to 946eab4 Compare July 13, 2016 20:56

tavinathanson merged commit 130b155 into master Jul 13, 2016

tavinathanson deleted the expressed_mutations branch July 13, 2016 22:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Filter to expressed variants #67

Filter to expressed variants #67

tavinathanson commented Jun 28, 2016

tavinathanson Jun 28, 2016

coveralls commented Jun 28, 2016 •

edited

arahuja Jun 28, 2016

tavinathanson Jun 28, 2016

tavinathanson Jun 28, 2016

arahuja commented Jun 28, 2016

tavinathanson commented Jun 28, 2016

tavinathanson commented Jun 28, 2016

tavinathanson commented Jun 28, 2016

arahuja commented Jun 29, 2016

tavinathanson commented Jun 29, 2016

coveralls commented Jul 9, 2016 •

edited

coveralls commented Jul 9, 2016 •

edited

coveralls commented Jul 10, 2016 •

edited

coveralls commented Jul 10, 2016 •

edited

tavinathanson commented Jul 10, 2016

coveralls commented Jul 10, 2016 •

edited

coveralls commented Jul 11, 2016 •

edited

coveralls commented Jul 12, 2016 •

edited

arahuja Jul 13, 2016

tavinathanson Jul 13, 2016

arahuja Jul 13, 2016

coveralls commented Jul 13, 2016 •

edited

coveralls commented Jul 13, 2016 •

edited

Filter to expressed variants #67

Filter to expressed variants #67

Conversation

tavinathanson commented Jun 28, 2016

tavinathanson Jun 28, 2016

Choose a reason for hiding this comment

coveralls commented Jun 28, 2016 • edited

arahuja Jun 28, 2016

Choose a reason for hiding this comment

tavinathanson Jun 28, 2016

Choose a reason for hiding this comment

tavinathanson Jun 28, 2016

Choose a reason for hiding this comment

arahuja commented Jun 28, 2016

tavinathanson commented Jun 28, 2016

tavinathanson commented Jun 28, 2016

tavinathanson commented Jun 28, 2016

arahuja commented Jun 29, 2016

tavinathanson commented Jun 29, 2016

coveralls commented Jul 9, 2016 • edited

coveralls commented Jul 9, 2016 • edited

coveralls commented Jul 10, 2016 • edited

coveralls commented Jul 10, 2016 • edited

tavinathanson commented Jul 10, 2016

coveralls commented Jul 10, 2016 • edited

coveralls commented Jul 11, 2016 • edited

coveralls commented Jul 12, 2016 • edited

arahuja Jul 13, 2016

Choose a reason for hiding this comment

tavinathanson Jul 13, 2016

Choose a reason for hiding this comment

arahuja Jul 13, 2016

Choose a reason for hiding this comment

coveralls commented Jul 13, 2016 • edited

coveralls commented Jul 13, 2016 • edited

coveralls commented Jun 28, 2016 •

edited

coveralls commented Jul 9, 2016 •

edited

coveralls commented Jul 9, 2016 •

edited

coveralls commented Jul 10, 2016 •

edited

coveralls commented Jul 10, 2016 •

edited

coveralls commented Jul 10, 2016 •

edited

coveralls commented Jul 11, 2016 •

edited

coveralls commented Jul 12, 2016 •

edited

coveralls commented Jul 13, 2016 •

edited

coveralls commented Jul 13, 2016 •

edited