Use align_method in comp_method_FRAME #22880

jbrockmendel · 2018-09-28T18:53:16Z

Closes #20090

update Since #23000 was merged, some of the discussion is out of date. The bottom line remains unchanged: This PR makes DataFrame comparison ops behave like DataFrame arithmetic ops currently do. Also fixes some bugs e.g. #20090
end update

This is a much nicer alternative to the implementation in #22751. The problem is that two tests still fail with this implementation. We need to pin down the design spec more explicitly regardless.

core.ops has three functions for defining DataFrame ops: _arith_method_FRAME, _flex_comp_method_FRAME, _comp_method_FRAME. The first two both call _align_method_FRAME, with _comp_method_FRAME being the outlier. This PR just adds that alignment call.

The two tests that currently fail:

df = DataFrame(np.arange(6).reshape((3, 2)))
b_r = np.atleast_2d([2, 2])
l = (2, 2, 2)
expected = DataFrame([[False, False], [False, True], [True, True]])
result = df > l   # <-- raises ValueError under this PR because it has the wrong shape
assert_frame_equal(result, expected)

result = df > b_r   # <-- raises ValueError under this PR because it has the wrong shape
assert_frame_equal(result, expected)

df = pd.DataFrame(np.arange(6).reshape((3, 2)))
with pytest.raises(ValueError):  # <-- doesnt raise
           df == (2, 2)

I understand why the behavior tested by the first test makes sense, but don't see the logic behind having df == (2, 2) raise (maybe #4576 holds the answer, will look at that more closely)

pep8speaks · 2018-09-28T18:53:23Z

Hello @jbrockmendel! Thanks for submitting the PR.

There are no PEP8 issues in the file pandas/core/frame.py !
There are no PEP8 issues in the file pandas/core/ops.py !
There are no PEP8 issues in the file pandas/tests/frame/test_operators.py !

jbrockmendel · 2018-09-28T18:55:47Z

Related: #22355.

jbrockmendel · 2018-09-29T16:28:05Z

@jorisvandenbossche this is the "pretty" alternative to #22751, needs attention despite not being green. Any thoughts on desired behavior?

jbrockmendel · 2018-10-03T21:56:17Z

@pandas-dev/pandas-core this now contains a non-trivial API change, should make sure there is consensus before moving forward.

TL;DR: broadcasting/alignment on DataFrame comparison ops behaves differently from arithmetic ops (#22355). Both behave differently from numpy convention (#22686). This PR changes the comparison behavior to match the arithmetic behavior, with the numpy part saved for the next round.

Demonstration (based on tests.frame.test_operators test_boolean_comparison in master):

df = pd.DataFrame(np.arange(6).reshape((3, 2)))
b = np.array([2, 2])
b_r = np.atleast_2d([2, 2])
b_c = b_r.T
l = (2, 2, 2)
tup = tuple(l)

cmp_expected = pd.DataFrame([[False, False], [False, True], [True, True]])
arith_expected = pd.DataFrame([[2, 3], [4, 5], [6, 7]])

# For the 1-D array `b`, comparison and arithmetic broadcast the same way
result = df > b
tm.assert_frame_equal(result, cmp_expected)
result = df.values > b
tm.assert_numpy_array_equal(result, cmp_expected.values)
result = df + b
tm.assert_frame_equal(result, arith_expected)
result = df.values + b
tm.assert_numpy_array_equal(result, arith_expected.values)

# In the 2-D case, the broadcasting behavior is different
result = df > b_r
tm.assert_frame_equal(result, cmp_expected)
result = df.values > b_r
tm.assert_numpy_array_equal(result, cmp_expected.values)

result = df.values + b_r
tm.assert_numpy_array_equal(result, arith_expected.values)

with pytest.raises(ValueError):  # we DONT want this to raise
    # ValueError: Unable to coerce to DataFrame, shape must be (3, 2): given (1, 2)
    result = df + b_r


# ditto for tuple/list
result = df > tup  # doesn't match numpy or arithmetic behavior
tm.assert_frame_equal(result, cmp_expected)

with pytest.raises(ValueError):
     # numpy behavior
     result = df.values > tup

with pytest.raises(ValueError):
     # numpy behavior
    result = df.values + tup

with pytest.raises(ValueError):
    # ValueError: Unable to coerce to Series, length must be 2: given 3
    result = df + tup

TomAugspurger · 2018-10-04T01:50:45Z

Just to make sure I understand, the changes would be

DataFrame + 2-d ndarray with matching trailing dimension: Previously raised, now broadcasts
DataFrame + non-ndarray iterable with matching first dimension: Previously broadcast, now raises

Did I miss any?

jbrockmendel · 2018-10-04T04:41:26Z

DataFrame + 2-d ndarray with matching trailing dimension: Previously raised, now broadcasts

~~np.ndarray with shape (1, df.shape[1]): previously broadcasts, now raises~~ Broadcasts following #23000

DataFrame + non-ndarray iterable with matching first dimension: Previously broadcast, now raises

list or tuple with len == df.shape[0] != df.shape[1]: previously broadcasts, now raises <-- AFAICT this behavior is not currently tested

list or tuple with len == df.shape[1]: previously raises, now broadcasts

* In a separate PR I would like to change the behavior when operating with ndarray with shape (nrows, 1) or (1, ncols) so that both of these broadcast identically to numpy (ATM the (nrows, 1) case raises for both arithmetic and comparison ops). This change would be for both arithmetic and comparison ops (whereas this PR just changes comparison ops to behave like arithmetic ops)

jbrockmendel · 2018-10-04T17:05:42Z

Update: In a different branch I edited the alignment code so that 2d arrays of (nrows, 1) or (1, ncols) both broadcast for arithmetic ops. This did not cause any test failures. So AFAICT this particular behavior may be unintentional.

update^2 #23000 has now implemented this behavior for arithmetic ops

codecov · 2018-10-04T19:07:50Z

Codecov Report

Merging #22880 into master will increase coverage by 0.03%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #22880      +/-   ##
==========================================
+ Coverage   92.21%   92.24%   +0.03%     
==========================================
  Files         169      169              
  Lines       50884    50814      -70     
==========================================
- Hits        46922    46874      -48     
+ Misses       3962     3940      -22

Flag	Coverage Δ
#multiple	`90.66% <100%> (+0.03%)`	⬆️
#single	`42.3% <100%> (+0.05%)`	⬆️

Impacted Files	Coverage Δ
pandas/core/internals/managers.py	`96.64% <ø> (-0.02%)`	⬇️
pandas/core/internals/blocks.py	`94.62% <ø> (+1.14%)`	⬆️
pandas/core/ops.py	`97.19% <100%> (ø)`	⬆️
pandas/core/frame.py	`97.11% <100%> (-0.01%)`	⬇️
pandas/io/formats/csvs.py	`98.21% <0%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e4b67ca...826f2c7. Read the comment docs.

jreback · 2018-10-06T15:58:37Z

you refernced #4576 which is indeed behavior we patched in the past, but doesn't necessary make sense for a DataFrame, while for a homogenous ndarray does a bit more.

Have to look at your changes in some detail..

jbrockmendel · 2018-10-06T17:35:40Z

Have to look at your changes in some detail..

Sure. The succinct version is my favorite: "Make DataFrame comparison ops behave like DataFrame arithmetic ops". That change is enabled entirely by the 1 line added in core.ops.

jbrockmendel · 2018-10-07T00:22:24Z

Extra benefit: After we get rid of Block.eval, it looks like _try_coerce_args can be simplified.

jbrockmendel · 2018-10-08T00:16:31Z

pandas/tests/frame/test_operators.py


-        result = df > b_r
+        result = df > b_r  # broadcasts like ndarray (GH#23000)


Changed following #23000 (previously in this PR this comparison raised ValueError)

can you put comments on the previous line

jbrockmendel · 2018-10-08T00:16:39Z

pandas/tests/frame/test_operators.py


-        result = df == b_r
+        result = df == b_r  # broadcasts like ndarray (GH#23000)


Changed following #23000 (previously in this PR this comparison raised ValueError)

comments on previous

TomAugspurger

There's a merge conflict with master now.

TomAugspurger · 2018-10-08T19:57:49Z

doc/source/whatsnew/v0.24.0.txt

+operations has been changed to match the arithmetic operations in these cases.
+(:issue:`22880`)
+
+The affected cases are: operating against a 2-dimensional ``np.ndarray`` with


Slight preference for formatting these cases as a list.

Just pushed addressing your comments. This one I had to guess what you had in mind. LMK if I got it wrong.

TomAugspurger · 2018-10-08T19:58:56Z

doc/source/whatsnew/v0.24.0.txt

+   In [3]: arr = np.arange(6).reshape(3, 2)
+   In [4]: df = pd.DataFrame(arr)
+
+   In [5]: df == arr[[0], :]  # comparison used to broadcast where arithmetic would raise


"used to" -> "previously" to avoid ambiguity with "used to".

TomAugspurger · 2018-10-08T20:01:18Z

doc/source/whatsnew/v0.24.0.txt

+   ...
+   ValueError: Unable to coerce to DataFrame, shape must be (3, 2): given (1, 2)
+
+   In [7]: df == (1, 2)  # length matches number of columns; comparison used to raise where arithmetic would broadcast


These code blocks are put in <pre> divs in the docs, so long lines will force horizontal scrolling / look extra bad on mobile. Better to continue the input with a

In [7]: ... ...: # comment ... ...: # maybe multiline

TomAugspurger · 2018-10-08T20:01:39Z

doc/source/whatsnew/v0.24.0.txt

+
+*Current Behavior*:
+
+.. ipython:: python


Will nee d an :okexcept here.

Can also remove the In [] prefixes. And hopefully the block continues running when there's an exception in an okexcept.

jreback

big issue I have here is what is the performance degredation. .eval dispatched to numexpr, I think that this is now handled in dispatch_to_* but not really sure. We may need some asvs to test the broadcasting behaviors (e.g. df + 2-d ndarray)

jreback · 2018-10-09T11:44:14Z

doc/source/whatsnew/v0.24.0.txt

+The affected cases are:
+  - operating against a 2-dimensional ``np.ndarray`` with either 1 row or 1 column
+  - a list or tuple with the same length matching the number of rows in the :class:`DataFrame`
+  - a list or tuple with thlength matching the number of columns in the :class:`DataFrame`.


jreback · 2018-10-09T11:45:03Z

doc/source/whatsnew/v0.24.0.txt

+   df = pd.DataFrame(arr)
+   df == arr[[0], :]  # raises just like the next arithmetic operation
+   df + arr[[0], :]
+   df == (1, 2)       # broadcasts just like the next arithmetic operation


would prefer that the comments precede the lines, also use multiple ipython blocks to make the rendering more clear (if needed).

jreback · 2018-10-09T11:45:52Z

pandas/tests/frame/test_operators.py


-        result = df > b_r
+        result = df > b_r  # broadcasts like ndarray (GH#23000)


can you put comments on the previous line

jreback · 2018-10-09T11:46:04Z

pandas/tests/frame/test_operators.py


-        result = df == b_r
+        result = df == b_r  # broadcasts like ndarray (GH#23000)


comments on previous

jorisvandenbossche · 2018-10-09T17:32:36Z

@jbrockmendel did you do a check of the performance to be sure there is no implication?

jbrockmendel · 2018-10-09T19:04:21Z

did you do a check of the performance to be sure there is no implication?

There does appear to be a performance hit in the relevant case, but a) there is room for subsequent optimization and b) the bugs this avoids are much more important than the performance impact.

In [3]: arr = np.arange(600).reshape(30, 20)
In [4]: df = pd.DataFrame(arr)
In [5] %timeit arr == arr[[0], :]    # calibrate based on numpy timings
10000 loops, best of 3: 82.9 µs per loop   <-- master
10000 loops, best of 3: 84 µs per loop      <-- PR

In [6]: ser = pd.Series(arr[0, :])
In [7]: l = list(ser)

In [8]: %timeit df == arr[[0], :]
100 loops, best of 3: 4.42 ms per loop   # <-- master
10 loops, best of 3: 87.5 ms per loop    # <-- PR

In [9]: %timeit df == ser
10 loops, best of 3: 83 ms per loop       <-- master
10 loops, best of 3: 86.9 ms per loop    <-- PR

In [10]: %timeit df == l
ValueError: Invalid broadcasting comparison       <-- master
10 loops, best of 3: 82.4 ms per loop    <-- PR

In [11]: df[1] = df[1].astype('f8')
    ...: df[3] = df[3].astype('uint8')
    ...: df[5] = df[5].view('m8[ns]')
    ...: df[7] = df[7].view('M8[ns]')
    ...: df[9] = df[9].astype('f4')
    ...:
In [12]: %timeit df == arr[[0], :]
ValueError: cannot broadcast shape [(30, 15)] with block values [(1, 20)]   <-- master, BUG
TypeError: cannot compare a TimedeltaIndex with type int64   <-- PR, BUG (in TimedeltaIndex.__eq__; just opened #23063)

In [13]: df[5] = df[5].view('i8').astype('i4')
In [14]: %timeit df == arr[[0], :]
ValueError: cannot broadcast shape [(30, 15)] with block values [(1, 20)]  <-- master, BUG
10 loops, best of 3: 86.7 ms per loop  <-- PR

In [14]: %timeit df == arr[0, :]
ValueError: Invalid broadcasting comparison [array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19])] with block values  <-- master
10 loops, best of 3: 80.9 ms per loop  <-- PR

jbrockmendel · 2018-10-12T15:54:22Z

gentle ping. If this (or #23031) goes through the follow-ups will keep me busy in places orthogonal to PeriodArray constructors

jorisvandenbossche · 2018-10-12T16:08:47Z

Quick general feedback: for the cases that worked before but will raise now, I think we should go through a deprecation cycle.

(looking now more in detail on your explanation of the changes above)

jorisvandenbossche · 2018-10-12T16:45:44Z

In [8]: %timeit df == arr[[0], :]
100 loops, best of 3: 4.42 ms per loop # <-- master
10 loops, best of 3: 87.5 ms per loop # <-- PR

How do you see this performance difference? I though this would now raise an error on this PR?

So if I understand correctly, some cases that now work, will raise after this PR, but you plan to let them work again in a later PR?
That makes it a bit difficult to assess what the overall changes for users would be (eg if a deprecation warning would be needed, see my previous post)

(it's not possible to for example first align arithmetic ops with numpy, and then comparison ops with arithmeric/numpy, so you don't have this back and forth?)

Something else to consider: for Index we actually deliberately disabled comparing with a length 1 array. See also the recently opened issue #23078 where we were planning to actually raise instead of broadcast

jorisvandenbossche · 2018-10-09T17:30:28Z

doc/source/whatsnew/v0.24.0.txt

+The affected cases are:
+  - operating against a 2-dimensional ``np.ndarray`` with either 1 row or 1 column
+  - a list or tuple with length matching the number of rows in the :class:`DataFrame`
+  - a list or tuple with length matching the number of columns in the :class:`DataFrame`.


this list is not valid rst formatted: you don't need to indentation, and need an empty line before the list

jorisvandenbossche · 2018-10-12T16:07:05Z

doc/source/whatsnew/v0.24.0.txt

+(:issue:`22880`)
+
+The affected cases are:
+  - operating against a 2-dimensional ``np.ndarray`` with either 1 row or 1 column


Can you indicate here what changes for those cases?

Will update

Updated; I think other whatsnew comments addressed. Note that Previous Behavior blocks refer to 0.23, not pre-this-PR

jbrockmendel · 2018-10-12T17:00:02Z

How do you see this performance difference?

If you drill down into this, it is driven by poor performance of cmp(DataFrame, Series), which I think can be improved on a subsequent pass. In the interim I see this as a uncommon corner case and the performance hit as being less important than internal consistency and bugfixes (just noticed this clsoes #20090)

I though this would now raise an error on this PR?

That was the case when this PR was opened, but since then #22355 changed the behavior of op(DataFrame, ndarray[ndim=2]) to match numpy broadcasting behavior.

(it's not possible to for example first align arithmetic ops with numpy, and then comparison ops with arithmeric/numpy, so you don't have this back and forth?)

If I understand correctly, this is similar to what I have in mind for optimizing the DataFrame vs Series op.

So if I understand correctly, some cases that now work, will raise after this PR, but you plan to let them work again in a later PR?

I think the only case that falls into this category is a list/tuple (not ndarray or Series) with length(obj) == len(df) != len(df.columns). Those comparisons currently "work" and this PR causes them to raise (matching arithmetic ops behavior).

I would be open to subsequently changing the behavior of both arithmetic+comparisons to do something like:

if len(obj) == len(df.columns):
    [behave like arithmetic ops currently do]
elif len(obj) == len(df):
    [behave like comparison ops currently do]

but only if this is made consistent across arithmetic/comparison ops.

jorisvandenbossche · 2018-10-12T17:04:58Z

I though this would now raise an error on this PR?

That was the case when this PR was opened, but since then #22355 changed the behavior of op(DataFrame, ndarray[ndim=2]) to match numpy broadcasting behavior.

But then the whatsnew addition you have in this PR is not up to date anymore? As that still mentions that this case will now raise an error

jbrockmendel · 2018-10-12T17:06:42Z

But then the whatsnew addition you have in this PR is not up to date anymore

Will update.

jorisvandenbossche · 2018-10-12T17:08:33Z

Can you please give a new summary of what this PR does / changes etc, as I am rather lost now. The comment above about "non-trivial API change" is also not relevant any more? (and update the top post)

jbrockmendel · 2018-10-12T17:21:59Z

Can you please give a new summary of what this PR does / changes etc, as I am rather lost now.

Updated above.

Bottom line: This PR makes DataFrame comparison ops broadcasting consistent with DataFrame arithmetic ops.

jorisvandenbossche · 2018-10-12T17:31:26Z

update Since #22355 was merged,

That's an open issue, not a merged PR?

jbrockmendel · 2018-10-12T18:42:35Z

Updated, #23000

jreback

lgtm. some minor comments. ping on green.

jreback · 2018-10-12T21:47:16Z

doc/source/whatsnew/v0.24.0.txt

+   df = pd.DataFrame(arr)
+
+.. ipython:: python
+   # comparison and arithmetic both broadcast


I'd prefer you make the comments actual sentences here, which is pretty easy when you already have the blocks.

jreback · 2018-10-12T21:47:55Z

pandas/tests/frame/test_arithmetic.py

@@ -48,15 +48,19 @@ def test_mixed_comparison(self):
        assert result.all().all()

    def test_df_boolean_comparison_error(self):
-        # GH 4576
+        # GH#4576, GH#22880
        # boolean comparisons with a tuple/list give unexpected results


can you change this comment

jreback · 2018-10-12T21:48:11Z

pandas/tests/frame/test_operators.py


-        result = df > tup
-        assert_frame_equal(result, expected)
+        with pytest.raises(ValueError):


can you assert the error messages in these

jbrockmendel · 2018-10-13T02:10:15Z

ping

jorisvandenbossche · 2018-10-13T07:58:35Z

@gfyoung sorry, but Brock and me were still actively discussing this. @jreback I think it was clear that was the case, so please don't say "ping on green" in such a case

This reverts commit e96c691.

jorisvandenbossche · 2018-10-13T08:01:07Z

OK, so I simply reverted the merge. That seemed the easiest way to continue the discussion. It's a bit annoying, but @jbrockmendel you will need to open a new PR since this one cannot be re-opened ..

Closes pandas-devgh-20090

…ndas-dev#23120) This reverts commit e96c691.

Use align_method in comp_method_FRAME

b6e3ed9

jbrockmendel mentioned this pull request Oct 2, 2018

[CLN] Dispatch (some) Frame ops to Series, avoiding _data.eval #22019

Merged

jbrockmendel added 3 commits October 3, 2018 14:04

Merge branch 'master' of https://github.com/pandas-dev/pandas into const

3baa31e

Remove no-longer-needed eval methods

aa9ca8f

Update tests to reflect consistent behavior

92513ca

jbrockmendel mentioned this pull request Oct 4, 2018

Make DataFrame arithmetic ops with 2D arrays behave like numpy analogues #23000

Merged

4 tasks

jbrockmendel added 2 commits October 5, 2018 10:37

whatsnew note

1090333

GH ref

fb5b16e

jbrockmendel added 2 commits October 7, 2018 17:09

Merge branch 'master' of https://github.com/pandas-dev/pandas into const

499e139

Adjust tested behavior following GH#23000

73cda15

jbrockmendel commented Oct 8, 2018

View reviewed changes

jbrockmendel changed the title ~~WIP: Use align_method in comp_method_FRAME~~ Use align_method in comp_method_FRAME Oct 8, 2018

jbrockmendel mentioned this pull request Oct 8, 2018

BUG: divmod return type #22932

Merged

TomAugspurger reviewed Oct 8, 2018

View reviewed changes

jbrockmendel added 2 commits October 8, 2018 15:49

Merge branch 'master' of https://github.com/pandas-dev/pandas into const

9c40569

Address Toms comments

87df9c4

jreback requested changes Oct 9, 2018

View reviewed changes

jreback added the Reshaping Concat, Merge/Join, Stack/Unstack, Explode label Oct 9, 2018

jorisvandenbossche reviewed Oct 12, 2018

View reviewed changes

update whatsnew

1d29857

jreback approved these changes Oct 12, 2018

View reviewed changes

jreback added this to the 0.24.0 milestone Oct 12, 2018

jbrockmendel added 3 commits October 12, 2018 15:17

Merge branch 'master' of https://github.com/pandas-dev/pandas into const

a3c8f21

fixup whatsnew, check specific exception messages

46a4e95

check messages compat for 32bit windows

826f2c7

gfyoung merged commit e96c691 into pandas-dev:master Oct 13, 2018

jorisvandenbossche mentioned this pull request Oct 13, 2018

Revert "Use align_method in comp_method_FRAME" #23120

Merged

jorisvandenbossche added a commit that referenced this pull request Oct 13, 2018

Revert "Use align_method in comp_method_FRAME (#22880)"

2f6d145

This reverts commit e96c691.

jorisvandenbossche added a commit that referenced this pull request Oct 13, 2018

Revert "Use align_method in comp_method_FRAME (#22880)" (#23120)

01cf59a

This reverts commit e96c691.

jbrockmendel mentioned this pull request Oct 13, 2018

Use align_method in comp_method_FRAME #23132

Merged

tm9k1 pushed a commit to tm9k1/pandas that referenced this pull request Nov 19, 2018

Use align_method in comp_method_FRAME (pandas-dev#22880)

3e7f099

Closes pandas-devgh-20090

tm9k1 pushed a commit to tm9k1/pandas that referenced this pull request Nov 19, 2018

Revert "Use align_method in comp_method_FRAME (pandas-dev#22880)" (pa…

3feaa79

…ndas-dev#23120) This reverts commit e96c691.

jbrockmendel deleted the const branch April 5, 2020 17:39


		result = df > b_r
		result = df > b_r # broadcasts like ndarray (GH#23000)


		result = df == b_r
		result = df == b_r # broadcasts like ndarray (GH#23000)

Use align_method in comp_method_FRAME #22880

Use align_method in comp_method_FRAME #22880

Conversation

jbrockmendel commented Sep 28, 2018 • edited Loading

pep8speaks commented Sep 28, 2018

jbrockmendel commented Sep 28, 2018

jbrockmendel commented Sep 29, 2018

jbrockmendel commented Oct 3, 2018

TomAugspurger commented Oct 4, 2018

jbrockmendel commented Oct 4, 2018 • edited Loading

jbrockmendel commented Oct 4, 2018 • edited Loading

codecov bot commented Oct 4, 2018 • edited Loading

Codecov Report

jreback commented Oct 6, 2018

jbrockmendel commented Oct 6, 2018

jbrockmendel commented Oct 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomAugspurger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche commented Oct 9, 2018

jbrockmendel commented Oct 9, 2018

jbrockmendel commented Oct 12, 2018

jorisvandenbossche commented Oct 12, 2018

jorisvandenbossche commented Oct 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbrockmendel commented Oct 12, 2018

jorisvandenbossche commented Oct 12, 2018

jbrockmendel commented Oct 12, 2018

jorisvandenbossche commented Oct 12, 2018 • edited Loading

jbrockmendel commented Oct 12, 2018

jorisvandenbossche commented Oct 12, 2018

jbrockmendel commented Oct 12, 2018

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbrockmendel commented Oct 13, 2018

jorisvandenbossche commented Oct 13, 2018

jorisvandenbossche commented Oct 13, 2018 • edited Loading

jbrockmendel commented Sep 28, 2018 •

edited

Loading

jbrockmendel commented Oct 4, 2018 •

edited

Loading

jbrockmendel commented Oct 4, 2018 •

edited

Loading

codecov bot commented Oct 4, 2018 •

edited

Loading

jorisvandenbossche commented Oct 12, 2018 •

edited

Loading

jorisvandenbossche commented Oct 13, 2018 •

edited

Loading