Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensure compatibility with numpy 2.0.0 #6976

Merged
merged 5 commits into from
Jun 19, 2024
Merged

Ensure compatibility with numpy 2.0.0 #6976

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
8575bbd
Ensure compatibility with numpy 2.0.0
KennethEnevoldsen Jun 17, 2024
4b72f46
Update src/datasets/formatting/formatting.py
KennethEnevoldsen Jun 18, 2024
3441b60
Merge branch 'main' into numpy2conversion
KennethEnevoldsen Jun 18, 2024
e53d6b9
Update src/datasets/formatting/formatting.py
KennethEnevoldsen Jun 18, 2024
5c532a9
make style
lhoestq Jun 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion src/datasets/formatting/formatting.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,14 +187,20 @@ def _arrow_array_to_numpy(self, pa_array: pa.Array) -> np.ndarray:
else:
zero_copy_only = _is_zero_copy_only(pa_array.type) and not _is_array_with_nulls(pa_array)
array: List = pa_array.to_numpy(zero_copy_only=zero_copy_only).tolist()

if len(array) > 0:
if any(
(isinstance(x, np.ndarray) and (x.dtype == object or x.shape != array[0].shape))
or (isinstance(x, float) and np.isnan(x))
for x in array
):
if np.lib.NumpyVersion(np.__version__) >= '2.0.0b1':
return np.array(array, dtype=object)
KennethEnevoldsen marked this conversation as resolved.
Show resolved Hide resolved
return np.array(array, copy=False, dtype=object)
return np.array(array, copy=False)
if np.lib.NumpyVersion(np.__version__) >= '2.0.0b1':
return np.array(array)
KennethEnevoldsen marked this conversation as resolved.
Show resolved Hide resolved
else:
return np.array(array, copy=False)


class PandasArrowExtractor(BaseArrowExtractor[pd.DataFrame, pd.Series, pd.DataFrame]):
Expand Down