-
-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add NumpyBatch utils from PVNet #287
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @markus-kreft, thanks very much for jumping on this. We really appreciate your help.
For the type-hint, I would use the NumpyBatch
class which can be imported from ocf_datapipes.batch
. Since we aim to support NumpyBatch with this function rather than any arbitrary nested dict
Hi @dfulu, thank you for the hint. Making the functions Batch-specific was also my first idea, but getting the types right also turns out to be quit difficult with the recursion (in the second call it will not be a NumpyBatch anymore), and allowing multiple types makes trouble when accessing the batch keys. To enforce the desired types in the public facing API I have two ideas:
def batch_to_tensor(batch: NumpyBatch) -> TensorBatch:
return _batch_to_tensor(batch)
From this SO post I think turning Batches into |
87b3380
to
43af752
Compare
for more information, see https://pre-commit.ci
I have implemented option 1 for now. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @markus-kreft, sorry for the delay. We've had a lot of internal deadlines at OCF in the last couple of weeks. The type hinting here is a quite difficult because of the recursion, but I think you've come up with a good pragmatic solution. So thanks for this work
@all-contributors please add @markus-kreft for code |
I've put up a pull request to add @markus-kreft! 🎉 |
Hi James, no worries, I can imagine you have a lot going on at OCF right now. I am happy my approach to typing worked out :-) I will integrate the functions in PVNet. |
Pull Request
Description
Transfer functions from PVNet specific to NumpyBatches here. Type hints are very minimal because recursive type support is still quite tricky. I have tried to use a nested dict type
NestedDict = dict[Any, Union[T, "NestedDict[T]"]]
but have trouble getting mypy to recognize NumpyBatches as such. I would be very interested if someone can teach me how to do this properly.Fixes openclimatefix/PVNet#111
How Has This Been Tested?
Logic comes for another repository. I have added a simple test in
tests/batch/test_utils.py
Checklist: