Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

added handling of object dtype in decode method data #2846

Merged
merged 2 commits into from
May 8, 2024
Merged

added handling of object dtype in decode method data #2846

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b678875
added handling of object dtype in decode method data
activesoull May 8, 2024
155e040
merged common logic
activesoull May 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 3 additions & 1 deletion deeplake/core/storage/gcs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -528,5 +528,7 @@ def get_object_from_full_url(self, url: str):

def get_creds(self):
d = self.scoped_credentials.get_token_info()
d["expiration"] = self.expiration or ""
d["expiration"] = (
self.expiration if hasattr(self, "expiration") and self.expiration else ""
)
return d
26 changes: 26 additions & 0 deletions deeplake/enterprise/dataloader.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -801,6 +801,29 @@ def __get_indra_dataloader(
info=info,
)

def _fill_sample_info_tensors(
self,
dataset,
sample_info_tensors,
json_tensors,
list_tensors,
):
for tensor_name in sample_info_tensors:
tensor = dataset._get_tensor_from_root(tensor_name)
if len(tensor) == 0:
raise EmptyTensorError(
f" the dataset has an empty tensor {tensor_name}, pytorch dataloader can't be created."
f" Please either populate the tensor or pass tensors argument to .pytorch that excludes this"
f" tensor."
)
meta = tensor.meta
if meta.htype == "json":
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why it's not one if with or-s?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've merged tags and list cases but the json needs to be in a different list due to deeplake decoding specifics

json_tensors.append(tensor_name)
elif meta.htype == "list":
list_tensors.append(tensor_name)
elif meta.htype == "tag":
list_tensors.append(tensor_name)

def __iter__(self):
if self._dataloader is None:
dataset = self.dataset
Expand All @@ -825,6 +848,9 @@ def __iter__(self):
sample_info_tensors, tensor_info_tensors = find_additional_tensors_and_info(
dataset, data_tensors
)
self._fill_sample_info_tensors(
dataset, sample_info_tensors, json_tensors, list_tensors
)
tensors.extend(sample_info_tensors)
htype_dict, ndim_dict, tensor_info_dict = get_htype_ndim_tensor_info_dicts(
dataset, data_tensors, tensor_info_tensors
Expand Down