Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Enable indexing with nullable Boolean #31591

Merged
merged 70 commits into from
Feb 22, 2020
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
75c915f
Add test
Feb 1, 2020
9c5b9f0
Remove test that checks for error
Feb 2, 2020
2441b40
Add frame test
Feb 3, 2020
d71d1ba
Don't raise with nullable boolean
Feb 3, 2020
4d3a264
Don't modify result
Feb 3, 2020
543ef9a
Add frame test
Feb 3, 2020
d3e7a69
Update whatsnew
Feb 3, 2020
ad7ae66
Fill NA
Feb 3, 2020
6991394
Merge branch 'master' into bool-idx
Feb 3, 2020
f6e9ce5
Remove some more tests
Feb 3, 2020
1234407
Delete another test
Feb 3, 2020
9b7e879
Use to_numpy
Feb 3, 2020
efdd29a
Update whatsnew
Feb 3, 2020
7fa36b6
Don't check for NA
Feb 3, 2020
b8e3d6b
Revert "Remove test that checks for error"
Feb 4, 2020
bc3fe3f
Update NA test
Feb 4, 2020
73ad221
Revert "Remove some more tests"
Feb 4, 2020
547d7bc
Update Categorical test
Feb 4, 2020
5649445
Update getitem tests
Feb 4, 2020
bb3d143
Update indexers.py
Feb 4, 2020
f107252
tm -> self
Feb 4, 2020
7b924b7
Assert for EA not DataFrame
Feb 4, 2020
46d77df
Don't try / except
Feb 4, 2020
ac71cbf
Change check_indexer test
Feb 4, 2020
e5ed092
Modify __getitem__ for datetimelike
Feb 4, 2020
9fcdb23
Add back ValueError for non-boolean with NA
Feb 4, 2020
c2dfa93
Revert "Delete another test"
Feb 4, 2020
a9a12b1
Fixup error message
Feb 4, 2020
7c10f33
Add before and after examples
Feb 5, 2020
cf3d60d
Get rid of some tests
Feb 5, 2020
157d8b9
Cast another way
Feb 5, 2020
250f228
Import
Feb 5, 2020
647f0f6
Don't import unused
Feb 5, 2020
6ccd96d
Merge branch 'master' into bool-idx
Feb 8, 2020
a9e73de
Update whatsnew
Feb 10, 2020
adc3075
Update boolean.rst
Feb 10, 2020
29ff823
check_array_indexer docstring
Feb 10, 2020
0a58605
Edit 1.1.0 whatsnew
Feb 10, 2020
b38a209
Add to indexing.rst
Feb 10, 2020
5088cbb
Add back index parameter
Feb 10, 2020
54efdd9
Add some True values in test
Feb 10, 2020
c6b81ed
Edit boolean.rst
Feb 10, 2020
67800c6
Add list back to check_array_indexer test
Feb 10, 2020
4c334f3
Merge branch 'master' into bool-idx
Feb 10, 2020
578fd3c
Account for pd.NA in is_bool_indexer
Feb 10, 2020
a559385
Include list mask in test
Feb 10, 2020
705947e
Account for empty key
Feb 10, 2020
4974778
Revert "Account for empty key"
Feb 10, 2020
319b525
Revert "Account for pd.NA in is_bool_indexer"
Feb 10, 2020
8007ce4
Try modifying is_bool_indexer
Feb 11, 2020
a10765f
Revert "Try modifying is_bool_indexer"
Feb 11, 2020
d7fc3b7
Revert "Include list mask in test"
Feb 11, 2020
bca582e
Merge branch 'master' into bool-idx
Feb 13, 2020
6f9a298
Update release notes and docs
Feb 13, 2020
e1e39fe
Add issue number to tests
Feb 13, 2020
5a72b2f
Add some setitem tests
Feb 13, 2020
c0e8dc7
Revert "Add some setitem tests"
Feb 13, 2020
a293bc6
Merge branch 'master' into bool-idx
Feb 13, 2020
607d9ed
Update setitem tests
Feb 13, 2020
2e7f9b3
Merge branch 'master' into bool-idx
Feb 15, 2020
bfe472b
Merge branch 'master' into bool-idx
Feb 16, 2020
a6294f8
Merge branch 'master' into bool-idx
Feb 16, 2020
c6d23f6
Add setitem test
Feb 17, 2020
c8ee434
Merge branch 'master' into bool-idx
Feb 17, 2020
fbda99d
Move whatsnew note
dsaxton Feb 19, 2020
3bf9327
Add back example
dsaxton Feb 19, 2020
dd65b0d
Merge branch 'master' into bool-idx
dsaxton Feb 19, 2020
974ec5d
Merge remote-tracking branch 'upstream/master' into bool-idx
dsaxton Feb 20, 2020
8f2d7bb
Merge remote-tracking branch 'upstream/master' into bool-idx
dsaxton Feb 21, 2020
080d1d2
Update comment
dsaxton Feb 21, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 8 additions & 40 deletions pandas/tests/extension/base/setitem.py
Original file line number Diff line number Diff line change
Expand Up @@ -98,8 +98,9 @@ def test_setitem_iloc_scalar_multiple_homogoneous(self, data):
[
np.array([True, True, True, False, False]),
pd.array([True, True, True, False, False], dtype="boolean"),
pd.array([True, True, True, pd.NA, pd.NA], dtype="boolean"),
],
ids=["numpy-array", "boolean-array"],
ids=["numpy-array", "boolean-array", "boolean-array-na"],
)
def test_setitem_mask(self, data, mask, box_in_series):
arr = data[:5].copy()
Expand All @@ -124,20 +125,17 @@ def test_setitem_mask_raises(self, data, box_in_series):
with pytest.raises(IndexError, match="wrong length"):
data[mask] = data[0]

def test_setitem_mask_boolean_array_raises(self, data, box_in_series):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rather than remove this can you turn it into a test (obviously changed that we no longer raise)

# missing values in mask
def test_setitem_mask_boolean_array_with_na(self, data, box_in_series):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this test duplicating the test_setitem_mask above?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Somewhat, yes

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can simply remove it then, I think?

mask = pd.array(np.zeros(data.shape, dtype="bool"), dtype="boolean")
mask[:2] = pd.NA
mask[:3] = True
mask[3:5] = pd.NA

if box_in_series:
data = pd.Series(data)

msg = (
"Cannot mask with a boolean indexer containing NA values|"
"cannot mask with array containing NA / NaN values"
)
with pytest.raises(ValueError, match=msg):
data[mask] = data[0]
data[mask] = data[0]

assert (data[:3] == data[0]).all()

@pytest.mark.parametrize(
"idx",
Expand Down Expand Up @@ -219,36 +217,6 @@ def test_setitem_mask_broadcast(self, data, setter):
assert ser[0] == data[10]
assert ser[1] == data[10]

def test_setitem_boolean_na_mask(self, data, box_in_series):
# https://github.com/pandas-dev/pandas/issues/31503
if box_in_series:
data = pd.Series(data)

mask = pd.array(np.zeros(len(data), dtype=bool), dtype="boolean")
mask[:2] = pd.NA
mask[2:4] = True
original = data.copy()

data[mask] = data[mask.fillna(False)]

if box_in_series:
tm.assert_series_equal(data, original)
else:
tm.assert_extension_array_equal(data, original)

def test_setitem_boolean_na_mask_frame(self, data):
# https://github.com/pandas-dev/pandas/issues/31503
df = pd.DataFrame({"a": range(len(data)), "b": data})
original = df.copy()

mask = pd.array(np.zeros(len(data), dtype=bool), dtype="boolean")
mask[:2] = pd.NA
mask[2:4] = True

df.loc[mask, "b"] = data[mask.fillna(False)]

tm.assert_frame_equal(df, original)

def test_setitem_expand_columns(self, data):
df = pd.DataFrame({"A": data})
result = df.copy()
Expand Down
4 changes: 0 additions & 4 deletions pandas/tests/extension/test_numpy.py
Original file line number Diff line number Diff line change
Expand Up @@ -415,10 +415,6 @@ def test_setitem_mask(self, data, mask, box_in_series):
def test_setitem_mask_raises(self, data, box_in_series):
super().test_setitem_mask_raises(data, box_in_series)

@skip_nested
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we have a setitem test?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I added a case containing NA to the the test_setitem_mask test Joris had merged previously (should be line 101 of setitem.py above)

def test_setitem_mask_boolean_array_raises(self, data, box_in_series):
super().test_setitem_mask_boolean_array_raises(data, box_in_series)

@skip_nested
@pytest.mark.parametrize(
"idx",
Expand Down