Example --> Pydantic

[peterdudfield]

Move Example to Pydantic object

Detailed Description

The Example data (maybe renamed to 'BatchData') is getting a bit bigger, and lacks a bit of structure. It might be good to move it to something with a bit more structure. If we need move to pydantic, then then we can use the

• internal validators to validate the data nicely.
• descriptions of data fields
• more tree like structure, which can grow with more data feeds

We would have to check that is saves and compresses well

Possible Implementation

BatchData

• General
  • t0_dt
  • batch_size
• PV
  • pv_yield
  • ....
• {Next_Data_Source}

[peterdudfield]

Would be interested in your opinions on this @JackKelly and @jacobbieker and how vital it is

[jacobbieker]

I don't know if its super urgent, but it sounds really nice to have the extra structure. After #150 there isn't more data sources that I can think of that are missing from what SatFlow had, other than cloud masks. But it would be good to have better structure for adding more sources as time goes on.

[JackKelly]

Sounds good!

We should check if PyTorch is happy to accept Pydantic objects (if not, I guess we can just convert from Pydantic to dicts). I'm not sure if PyTorch will be happy with nested dicts (but I haven't tried!)

Also, at the moment, Example objects are (confusingly!) used to hold two different things:

1. Individual examples
2. Entire batches (i.e. multiple examples)

I'm not sure how best to rename Example to a name that's general enough to capture these two uses.

And, at different stages of the pipeline, each field might be a pd.DataFrame or xr.DataArray, a numpy array, or a Tensor :) It would be good to try to make this more explicit in the code, so we always know exactly which data types we're dealing with.

[jacobbieker]

For being more explicit what each data type is throughout the code, I've used schema for that, which worked quite well, or apparently, PyDantic has a beta validator as well we could use?

[peterdudfield]

Yea, definately are a few different things going on.

I was thinking of not passing pydantic objects to PyTorch, just the individual elements.

Perhaps we could have

1. Individual examples
2. Batch

We could then have methods in the objects, that maybe converts to xr.DataArrays, or to numpys

Pydantic validators, are around already,
https://pydantic-docs.helpmanual.io/usage/validators/
But I see the how 'validation_decorator' could be useful. We can always use a few asserts to check things for the moment

[peterdudfield]

Oh and pydantic objects already have a basical validator in the for for example

from pydantic import BaseModel
class OneGSP(BaseModel):
    gsp_id: int

g = OneGSP(gsp_id='test_string') 

• causes an error

gsp_id value is not a valid integer (type=type_error.integer)

[JackKelly]

Very nice!

[peterdudfield]

https://github.com/openclimatefix/nowcasting_dataset/blob/issue/166-batch-pydantic/notebooks/2021-10/2021-10-01/pydantic.py

I did a little test to see if pytorch could handle pydantic objects - it could not, but it could handle nested dictionarys

[peterdudfield]

https://github.com/openclimatefix/nowcasting_dataset/tree/issue/166-batch-pydantic/nowcasting_dataset/dataset/model
Ive put a general model, plus a satellite model together. Obviously there is more to add.

I think having it like this will make things really nice, and hopefully easier to work with

I would be interested in some intermediate feedback - rather than doing it all and getting it all wrong. (@JackKelly @jacobbieker )