New Orientation Bias Filter #4895

takutosato · 2018-06-13T16:22:48Z

The WGS evaluation shows that the new filter is equivalent to or slightly better than the old filter for filtering the known Oxo-G and FFPE base artifacts. The new filter is strictly better at filtering artifacts we haven't encountered before, like that of the Novaseq data.

This PR also introduces the FDR-based two-pass filtering for FilterMutectCalls #4893

@davidbenjamin I still need to revise mutect2.tex, but the code is ready for review

@LeeTL1220 @madduran

davidbenjamin

I still need to review LearnReadOrientationModelEngine and the docs but here are comments on everything else to start with.

davidbenjamin · 2018-06-13T17:28:27Z

docs/mutect/mutect.tex

+
+Let $z_i$ denote a 1-hot encoding of the genotype extended to include the possibility of read orientation-based artifact at locus $i$. For example, at a locus where the 3 base-pair reference context is ACT, we have
+
+$$z \in \{ \text{F1R2}_A, \text{F1R2}_G, \text{F1R2}_T, \text{F2R1}_A, \text{F2R1}_G, \text{F2R1}_T,  \text{Hom Ref}, \text{Germline Het}, \text{Somatic Het}, \text{Hom Var} \}$$


Use \begin{equation}

davidbenjamin · 2018-06-13T17:29:50Z

docs/mutect/mutect.tex

+
+$z_i$ has an associated vector of prior probabilities $\pi$.
+
+Assignment to $z_i$ dictates hyperparameters $\alpha, \beta$ for the conditional distribution over the number of alt reads, $m_i | z_i \sim \mathrm{BetaBinomial}(n, \alpha, \beta)$, where $n_i$ is the depth at site $i$. The number of F1R2 alt reads, denoted $x_i$, is also a beta-binomial distribution with parameters $\alpha'$ and $\beta'$ that depend on $z_i$, but it draws out of $m$ alt reads.


Assignment to $z_i$ dictates -> $z_i$ determines

davidbenjamin · 2018-06-13T17:38:52Z

src/main/java/org/broadinstitute/hellbender/engine/ReferenceContext.java

+     *
+     * Returns null if at the end of a contig we cannot expand the window
+     */
+    public String getKmerAround(final int center, final int numBasesOnEachSide){


It seems like some code that calls this method doesn't handle the possibility of returning null.

Fixed (but not in tests)

davidbenjamin · 2018-06-13T17:41:32Z

...main/java/org/broadinstitute/hellbender/tools/walkers/annotator/ReadOrientationArtifact.java

+ * Created by tsato on 10/20/17.
+ */
+public class ReadOrientationArtifact extends GenotypeAnnotation implements StandardMutectAnnotation {
+    private File artifactPriorTable;


This doesn't need to be a member variable.

davidbenjamin · 2018-06-13T17:44:31Z

...main/java/org/broadinstitute/hellbender/tools/walkers/annotator/ReadOrientationArtifact.java

+
+    @Override
+    public List<VCFFormatHeaderLine> getDescriptions() {
+        return Arrays.asList(new VCFFormatHeaderLine(GATKVCFConstants.ROF_POSTERIOR_KEY, 1,


Put these in GATKVCFHeaderLines so that you can use GATKVCFHeaderLines.getFormatLine(KEY) here.

davidbenjamin · 2018-06-20T18:30:25Z

src/main/java/org/broadinstitute/hellbender/utils/MathUtils.java

+     * @param n number of coin flips
+     * @return probability density function evaluated at k
+     */
+    public static double log10BetaBinomialProbability(final int k, final int n, final double alpha, final double beta){


In light of the last comment, just checking that this is intentional code for this PR and not a rebasing artifact.

Yup all is well I believe

davidbenjamin · 2018-06-20T18:33:10Z

src/main/java/org/broadinstitute/hellbender/utils/variant/GATKVCFHeaderLines.java

@@ -60,6 +59,9 @@ private static void addFilterLine(final VCFFilterHeaderLine line) {
        addFilterLine(new VCFFilterHeaderLine(READ_POSITION_FILTER_NAME, "median distance of alt variants from end of reads"));
        addFilterLine(new VCFFilterHeaderLine(CONTAMINATION_FILTER_NAME, "contamination"));
        addFilterLine(new VCFFilterHeaderLine(DUPLICATED_EVIDENCE_FILTER_NAME, "evidence for alt allele is overrepresented by apparent duplicates"));
+        addFilterLine(new VCFFilterHeaderLine(F1R2_ARTIFACT_FILTER_NAME, "over-representation of F1R2 reads among alt"));


I would say that it's an F1R2 orientation bias artifact instead, because just saying over-representation could be interpreted to mean that you just look at F1R2 and F2R1 counts without learning priors. We don't want users to jump to bad conclusions about the model.

davidbenjamin · 2018-06-20T18:34:07Z

src/main/python/org/broadinstitute/hellbender/gcnvkernel/models/model_denoising_calling.py

@@ -927,7 +927,7 @@ class HHMMClassAndCopyNumberBasicCaller:

        - Given q(c_s), the emission probability of each copy number class (tau) is determined
          (see _get_update_log_class_emission_tk_theano_func). The class prior and transition probabilities
-          are fixed hyperparameters. Therefore, q(tau) can be updated immediately using a single run
+          are fixed artifactPrior. Therefore, q(tau) can be updated immediately using a single run


IntelliJ was overzealous here.

davidbenjamin · 2018-06-20T18:35:29Z

...a/org/broadinstitute/hellbender/tools/walkers/annotator/ReadOrientationArtifactUnitTest.java

+    public void test(final int depth, final double alleleFraction, final double altF1R2Fraction,
+                     final double expectedArtifactProb, final double epsilon,
+                     final Optional<ReadOrientation> expectedArtifactType) throws IOException {
+        // Why is it that the annotations get larger window than the walker (CollectData?)


What's this comment about?

I think it can safely be ignored. Removed

davidbenjamin · 2018-06-20T18:38:08Z

...a/org/broadinstitute/hellbender/tools/walkers/readorientation/CollectF1R2CountsUnitTest.java

+/**
+ * Created by tsato on 3/14/18.
+ */
+public class CollectF1R2CountsUnitTest extends CommandLineProgramTest {


IntegrationTest

davidbenjamin

I have reviewed everything now.

davidbenjamin · 2018-06-25T00:23:28Z

docs/mutect/mutect.tex

+
+\section{Read Orientation Artifact Filter}
+
+Read orientation artifact, also known as orientation bias, arises due to a chemical change in the nucleotide during library prep that results in, for example, G base-paring with A. The artifact is context-specific (e.g. C to A occurring only when the surrounding reference context is CCG) and therefore single-stranded. Downstream, this artifact manifests as low allele fraction SNPs whose evidence for the alt allele consists almost entirely of one read orientation i.e. F1R2 or F2R1 with the similar signature repeated across the sam/bam/cram.


"Only" is too strong. Perhaps "predominantly"?

Define F1R2 and F2R1.

davidbenjamin · 2018-06-25T00:26:18Z

docs/mutect/mutect.tex

+\begin{equation*}
+z_i \in \{ \text{F1R2}_A, \text{F1R2}_G, \text{F1R2}_T, \text{F2R1}_A, \text{F2R1}_G, \text{F2R1}_T,  \text{Hom Ref}, \text{Germline Het}, \text{Somatic Het}, \text{Hom Var} \}
+\end{equation*}
+


This space shouldn't be here because it indents the following line.

davidbenjamin · 2018-06-25T00:26:45Z

docs/mutect/mutect.tex

+z_i \in \{ \text{F1R2}_A, \text{F1R2}_G, \text{F1R2}_T, \text{F2R1}_A, \text{F2R1}_G, \text{F2R1}_T,  \text{Hom Ref}, \text{Germline Het}, \text{Somatic Het}, \text{Hom Var} \}
+\end{equation*}
+
+where $z_i = \rm{F1R2_A}$ denotes that at locus $i$ we have a context specific artifact in which the evidence for alt allele A consists entirely of reads in the F1R2 orientation. The rest of artifact states are defined analogously. Let $\vpi$ denote the prior probabilities of the $\vz_i$ under the reference context ACT. Then we have 


"Rest of" --> "other" or "remaining"

davidbenjamin · 2018-06-25T00:27:17Z

docs/mutect/mutect.tex

+P(\vz_i) = \prod_k \pi_{k}^{z_{ik}}
+\end{equation}
+
+The number of alt reads at a locus is dependent on the genotype $z_i$. Let $n_i$ and $m_i$ denote the total depth and alt depth at locus $i$, respectively. The conditional distribution of $m_i$ is


"is dependent on" --> "depends on"

davidbenjamin · 2018-06-25T00:27:55Z

docs/mutect/mutect.tex

+\begin{equation}
+P(m_i | z_i) = \rm{BetaBinomial}(m_i | n_i, \alpha_{z_i}, \beta_{z_i})
+\end{equation}
+


As above, this space causes unwanted indentation.

davidbenjamin · 2018-06-25T03:12:06Z

...broadinstitute/hellbender/tools/walkers/readorientation/LearnReadOrientationModelEngine.java

+        ArtifactState.getF2R1ArtifactStates().forEach(s -> alleleFractionPseudoCounts.put(s, new Pair<>(altPseudocount, refPseudocount)));
+
+
+        for (ArtifactState z : ArtifactState.getNonArtifactStates()){


Can this be four puts instead of a switch?

davidbenjamin · 2018-06-25T03:12:37Z

...broadinstitute/hellbender/tools/walkers/readorientation/LearnReadOrientationModelEngine.java

+
+    private static Map<ArtifactState, Pair<Double, Double>> getPseudoCountsForAltF1R2Fraction(){
+        final Map<ArtifactState, Pair<Double, Double>> altF1R2FractionPseudoCounts = new HashMap<>(ArtifactState.values().length);
+        final double pseudoCountOfLikelyOutcome = 100.0;


Extract constants.

davidbenjamin · 2018-06-25T03:21:56Z

...broadinstitute/hellbender/tools/walkers/readorientation/LearnReadOrientationModelEngine.java

+        final Map<ArtifactState, Pair<Double, Double>> altF1R2FractionPseudoCounts = new HashMap<>(ArtifactState.values().length);
+        final double pseudoCountOfLikelyOutcome = 100.0;
+        final double pseudoCountOfRareOutcome = 1.0;
+        final double balancedPrior = 10;


I would think this could be a lot sharper -- Beta(10, 10) has a standard deviation of about 0.1.

davidbenjamin · 2018-06-25T03:22:58Z

...broadinstitute/hellbender/tools/walkers/readorientation/LearnReadOrientationModelEngine.java

+        return altF1R2FractionPseudoCounts;
+    }
+
+    public double[] getEffectiveCounts(){


This is unused according to IntelliJ.

It's now used in a unit test

davidbenjamin · 2018-06-25T03:24:29Z

...broadinstitute/hellbender/tools/walkers/readorientation/LearnReadOrientationModelEngine.java

+    private Triple<Integer, Nucleotide, ReadOrientation> createKey(final int depth, final Nucleotide altAllele, final ReadOrientation orientation){
+        return new ImmutableTriple<>(depth, altAllele, orientation);
+    }
+


takutosato · 2018-07-05T22:45:44Z

@davidbenjamin back to you. Rebasing got a bit complicated so I'm going to put that off until tomorrow.

davidbenjamin

Just a few more small things.

davidbenjamin · 2018-07-25T15:48:23Z

scripts/mutect2_wdl/mutect2.wdl

+        export GATK_LOCAL_JAR=${default="/root/gatk.jar" gatk_override}
+
+        gatk --java-options "-Xmx${command_mem}m" CollectF1R2Counts \
+        -I ${tumor_bam} -O "${tumor_sample_name}-artifact-prior-table.tsv" -R ${ref_fasta} \


indent these lines

davidbenjamin · 2018-07-25T15:49:40Z

scripts/mutect2_wdl/mutect2.wdl

+        bootDiskSizeGb: 12
+        memory: machine_mem + " MB"
+        disks: "local-disk " + select_first([disk_space, 100]) + if use_ssd then " SSD" else " HDD"
+        preemptible: select_first([preemptible_attempts, 10])


If my maxRetries PR gets in first, you'll need to add that to the runtime block. When you rebase you'll see how the other tasks do it.

davidbenjamin · 2018-07-25T15:49:51Z

scripts/mutect2_wdl/mutect2.wdl

+        export GATK_LOCAL_JAR=${default="/root/gatk.jar" gatk_override}
+
+        gatk --java-options "-Xmx${command_mem}m" LearnReadOrientationModel \
+        -R ${ref_fasta} \


davidbenjamin · 2018-07-25T15:50:18Z

scripts/mutect2_wdl/mutect2.wdl

+        export GATK_LOCAL_JAR=${default="/root/gatk.jar" gatk_override}
+
+        gatk --java-options "-Xmx${command_mem}m" LearnReadOrientationModel \
+        -R ${ref_fasta} \


Do the intervals get used here?

No, removed

davidbenjamin · 2018-07-25T15:50:29Z

scripts/mutect2_wdl/mutect2.wdl

+        bootDiskSizeGb: 12
+        memory: machine_mem + " MB"
+        disks: "local-disk " + select_first([disk_space, 100]) + if use_ssd then " SSD" else " HDD"
+        preemptible: select_first([preemptible_attempts, 10])


same about maxRetries

davidbenjamin · 2018-07-25T16:01:29Z

...main/java/org/broadinstitute/hellbender/tools/walkers/annotator/ReadOrientationArtifact.java

-
-            // Throw away bases that is below the minimum quality
-            if (asm.getGenomePosition() == vc.getStart() && read.getBaseQuality(readOffset) < minimumBaseQuality){
+            // Throw away bases that is below the desired minimum quality


davidbenjamin · 2018-07-25T19:39:17Z

...ava/org/broadinstitute/hellbender/tools/walkers/readorientation/ArtifactPriorCollection.java

+         */
+        for (final String refContext : F1R2FilterConstants.CANONICAL_KMERS){
+            final Optional<ArtifactPrior> ap = priors.stream().filter(a -> a.getReferenceContext().equals(refContext)).findAny();
+            Utils.validate(ap.isPresent(), "ArtifactPrior object isn't present for reference context " + refContext + "in file " + input);


This would be better as a UserException.BadInput

davidbenjamin · 2018-07-25T19:40:15Z

.../java/org/broadinstitute/hellbender/tools/walkers/readorientation/BetaDistributionShape.java

+    private double beta;
+
+    public BetaDistributionShape(final double alpha, final double beta){
+        Utils.validateArg(alpha > 0 , "alpha must be greater than 0 but got " + beta);


ParamUtils.isPositive is more precise.

davidbenjamin · 2018-07-25T19:43:48Z

...broadinstitute/hellbender/tools/walkers/readorientation/LearnReadOrientationModelEngine.java

+    private final static double REF_PSEUDOCOUNT = 9.0;
+
+    // for home ref sites assume Q35, but maintaining some width
+    private final static double PSEUDO_COUNT_OF_HOM_LIKELY = 10000.0;


PSEUDO_COUNT --> PSEUDOCOUNT

davidbenjamin · 2018-07-25T19:45:18Z

Back to @takutosato. You're almost done.

takutosato · 2018-07-27T20:20:24Z

@davidbenjamin I made the requested changes and submitted novaseq validation jobs. They haven't failed yet, but I'll monitor the jobs and make changes to the wdl as needed. Will let you know when they finish.

davidbenjamin · 2018-07-29T03:53:29Z

@takutosato You can merge once tests pass.

codecov-io · 2018-07-31T20:27:23Z

Codecov Report

Merging #4895 into master will increase coverage by 0.227%.
The diff coverage is 93.158%.

@@               Coverage Diff               @@
##              master     #4895       +/-   ##
===============================================
+ Coverage     86.387%   86.614%   +0.227%     
- Complexity     28815     29833     +1018     
===============================================
  Files           1791      1813       +22     
  Lines         133696    137705     +4009     
  Branches       14896     15619      +723     
===============================================
+ Hits          115496    119272     +3776     
- Misses         12797     12889       +92     
- Partials        5403      5544      +141

Impacted Files	Coverage Δ	Complexity Δ
...nstitute/hellbender/engine/filters/ReadFilter.java	`100% <ø> (ø)`	`10 <0> (ø)`	⬇️
...ute/hellbender/utils/variant/GATKVCFConstants.java	`80% <ø> (ø)`	`4 <0> (ø)`	⬇️
...ion/basicshortmutpileup/PowerCalculationUtils.java	`92.683% <0%> (-3.984%)`	`31 <0> (+13)`
...hellbender/tools/walkers/mutect/Mutect2Engine.java	`93.396% <0%> (+2.005%)`	`71 <1> (+16)`	⬆️
...ientation/ReadOrientationModelIntegrationTest.java	`100% <100%> (ø)`	`6 <6> (?)`
...dinstitute/hellbender/utils/MathUtilsUnitTest.java	`92.369% <100%> (+0.21%)`	`143 <4> (+3)`	⬆️
...ools/walkers/annotator/OxoGReadCountsUnitTest.java	`100% <100%> (ø)`	`15 <0> (ø)`	⬇️
...e/hellbender/utils/variant/GATKVCFHeaderLines.java	`99.329% <100%> (+0.028%)`	`10 <0> (ø)`	⬇️
...walkers/readorientation/AltSiteRecordUnitTest.java	`100% <100%> (ø)`	`10 <10> (?)`
.../hellbender/tools/walkers/mutect/FilterResult.java	`100% <100%> (ø)`	`7 <7> (?)`
... and 94 more

ldgauthier · 2018-08-03T17:36:40Z

Woohoo!

…

On Fri, Aug 3, 2018 at 12:44 PM, Takuto Sato ***@***.***> wrote: Merged #4895 <#4895> into master. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4895 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGRhdCSDz1_4UPPxztU6NZMIWkGFXdcfks5uNH3ZgaJpZM4Umhce> .

-- Laura Doyle Gauthier, Ph.D. Associate Director, Germline Methods Data Sciences Platform gauthier@broadinstitute.org Broad Institute of MIT & Harvard 320 Charles St. Cambridge MA 0214

…uplicated code in tests surrounding Mutect2.

…uplicated code in tests surrounding Mutect2. (#5123)

takutosato requested a review from davidbenjamin June 14, 2018 19:38

takutosato assigned davidbenjamin Jun 14, 2018

davidbenjamin requested changes Jun 20, 2018

View reviewed changes

davidbenjamin requested changes Jun 25, 2018

View reviewed changes

takutosato force-pushed the ts_ob_2pass branch 2 times, most recently from 8d8a0ad to 882cd93 Compare July 9, 2018 16:40

takutosato mentioned this pull request Jul 23, 2018

Data collection part of the read orientation filter #3721

Closed

davidbenjamin requested changes Jul 25, 2018

View reviewed changes

davidbenjamin assigned takutosato and unassigned davidbenjamin Jul 25, 2018

takutosato force-pushed the ts_ob_2pass branch 2 times, most recently from da41ad6 to 409a20c Compare July 27, 2018 20:15

davidbenjamin approved these changes Jul 29, 2018

View reviewed changes

New Orientation Bias Filter

14c6d94

takutosato force-pushed the ts_ob_2pass branch from 12d6db6 to 14c6d94 Compare August 3, 2018 15:33

takutosato merged commit 1acffc1 into master Aug 3, 2018

takutosato mentioned this pull request Aug 7, 2018

Read Orientation Model test case where the variant is no longer emitted in the vcf #5091

Merged

davidbenjamin mentioned this pull request Aug 9, 2018

Generative model for orientation bias #3016

Closed

takutosato added a commit that referenced this pull request Aug 20, 2018

restore base quality filter code that got reverted in #4895

17c7fb5

takutosato mentioned this pull request Aug 20, 2018

Restore base quality filter code that got reverted in #4895 #5123

Merged

takutosato added a commit that referenced this pull request Aug 20, 2018

restore base quality filter code that got reverted in #4895

8ee28fd

chapmanb mentioned this pull request Aug 23, 2018

String index out of range in ReferenceBases.annotate when calling variants at ends of chromosomes using MuTect2 #5130

Closed

takutosato added a commit that referenced this pull request Aug 29, 2018

Restore base quality filter code that got reverted in #4895. Reduce d…

2acc809

…uplicated code in tests surrounding Mutect2.

takutosato added a commit that referenced this pull request Aug 30, 2018

Restore base quality filter code that got reverted in #4895. Reduce d…

9df7b59

…uplicated code in tests surrounding Mutect2.

takutosato mentioned this pull request Aug 30, 2018

Fix Edge Cases in Reference Bases #5151

Merged

takutosato added a commit that referenced this pull request Aug 31, 2018

Restore base quality filter code that got reverted in #4895. Reduce d…

9fb2a65

…uplicated code in tests surrounding Mutect2. (#5123)

davidbenjamin deleted the ts_ob_2pass branch February 24, 2021 15:06


		Let $z_i$ denote a 1-hot encoding of the genotype extended to include the possibility of read orientation-based artifact at locus $i$. For example, at a locus where the 3 base-pair reference context is ACT, we have

		$$z \in \{ \text{F1R2}_A, \text{F1R2}_G, \text{F1R2}_T, \text{F2R1}_A, \text{F2R1}_G, \text{F2R1}_T, \text{Hom Ref}, \text{Germline Het}, \text{Somatic Het}, \text{Hom Var} \}$$


		$z_i$ has an associated vector of prior probabilities $\pi$.

		Assignment to $z_i$ dictates hyperparameters $\alpha, \beta$ for the conditional distribution over the number of alt reads, $m_i \| z_i \sim \mathrm{BetaBinomial}(n, \alpha, \beta)$, where $n_i$ is the depth at site $i$. The number of F1R2 alt reads, denoted $x_i$, is also a beta-binomial distribution with parameters $\alpha'$ and $\beta'$ that depend on $z_i$, but it draws out of $m$ alt reads.


		\section{Read Orientation Artifact Filter}

		Read orientation artifact, also known as orientation bias, arises due to a chemical change in the nucleotide during library prep that results in, for example, G base-paring with A. The artifact is context-specific (e.g. C to A occurring only when the surrounding reference context is CCG) and therefore single-stranded. Downstream, this artifact manifests as low allele fraction SNPs whose evidence for the alt allele consists almost entirely of one read orientation i.e. F1R2 or F2R1 with the similar signature repeated across the sam/bam/cram.

		ArtifactState.getF2R1ArtifactStates().forEach(s -> alleleFractionPseudoCounts.put(s, new Pair<>(altPseudocount, refPseudocount)));


		for (ArtifactState z : ArtifactState.getNonArtifactStates()){

New Orientation Bias Filter #4895

New Orientation Bias Filter #4895

Conversation

takutosato commented Jun 13, 2018

davidbenjamin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidbenjamin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

takutosato commented Jul 5, 2018

davidbenjamin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidbenjamin commented Jul 25, 2018

takutosato commented Jul 27, 2018

davidbenjamin commented Jul 29, 2018

codecov-io commented Jul 31, 2018 • edited

Codecov Report

ldgauthier commented Aug 3, 2018 via email

codecov-io commented Jul 31, 2018 •

edited