Fix splitter errors for datasets without labels #2641
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@rbharath I feel there might be a better way to do this, but for now this works as intended with the USPTO loader. |
rbharath
reviewed
Aug 9, 2021
deepchem/data/tests/test_shape.py
Outdated
| w = np.random.randint(2, size=(num_datapoints, num_tasks)) | ||
| ids = np.array(["id"] * num_datapoints) | ||
|
|
||
| dataset = dc.data.DiskDataset.from_numpy(X, y, w, ids) |
There was a problem hiding this comment.
So here I'm thinking more if we do
dataset = dc.data.DiskDataset.from_numpy(X)
assert dataset.get_shard_size() == 100
The idea is we want to check that datasets without labels work correctly :)
There was a problem hiding this comment.
yup! will add that
rbharath
reviewed
Aug 12, 2021
|
|
||
| Note | ||
| ---- | ||
| DiskDatasets without labels cannot be resharded! |
There was a problem hiding this comment.
This should be fixed! Can you raise a separate issue for this and we can get a fix going?
|
@rbharath Could you try restarting the checks maybe? I think I opened the PR when github actions was down. |
rbharath
approved these changes
Aug 23, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Some datasets without labels such as USPTO, Swiss-Prot are not compatible with the splitters as the get_shard_size method looks for the length of y, which in this case is None. This is a crack at getting around this issue, I was able to load in and split the USPTO datasets with this.
Type of change
Please check the option that is related to your PR.
Checklist
yapf -i <modified file>and check no errors (yapf version must be 0.22.0)mypy -p deepchemand check no errorsflake8 <modified file> --countand check no errors