Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BatchFeature: Convert List[np.ndarray] to np.ndarray before converting to pytorch tensors #14306

Merged
merged 7 commits into from
Nov 10, 2021
Merged

BatchFeature: Convert List[np.ndarray] to np.ndarray before converting to pytorch tensors #14306

Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
68e3756
update
eladsegal Nov 7, 2021
05fed8b
style fix
eladsegal Nov 7, 2021
1f69c7f
retrigger checks
eladsegal Nov 8, 2021
5e3fbae
check first element
eladsegal Nov 8, 2021
a3cc3fc
fix syntax error
eladsegal Nov 8, 2021
c6f43b5
Update src/transformers/feature_extraction_utils.py
eladsegal Nov 10, 2021
a699a36
remove import
eladsegal Nov 10, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion src/transformers/feature_extraction_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
import json
import os
from collections import UserDict
from collections.abc import Sequence
from typing import TYPE_CHECKING, Any, Dict, Optional, Tuple, Union

import numpy as np
Expand Down Expand Up @@ -138,7 +139,13 @@ def convert_to_tensors(self, tensor_type: Optional[Union[str, TensorType]] = Non
raise ImportError("Unable to convert output to PyTorch tensors format, PyTorch is not installed.")
import torch

as_tensor = torch.tensor
def as_tensor(value):
if isinstance(value, Sequence) and len(value) > 0:
first_elem = value[0]
if isinstance(first_elem, np.ndarray):
value = np.array(value)
eladsegal marked this conversation as resolved.
Show resolved Hide resolved
return torch.tensor(value)

is_tensor = torch.is_tensor
elif tensor_type == TensorType.JAX:
if not is_flax_available():
Expand Down