Move all data normalisation to nowcasting_dataloader

[peterdudfield]

Detailed Description

Currently satellite normalisation can happen in two places

• https://github.com/openclimatefix/nowcasting_dataset/blob/main/nowcasting_dataset/data_sources/satellite/satellite_data_source.py#L186
• https://github.com/openclimatefix/nowcasting_dataset/blob/main/nowcasting_dataset/dataset/datasets.py#L210

Context

• Dont want to do it twice, as this may lead to funny ML problems
• Would be nice if the metadata stored if the data had been normalised or not.
• Nice to have it un normalised in batches too, so it can be displayed easily

Possible Implementation

Only have option to do it once.

• It could be a submethod of one of the Satellite classes.
• The metadata stored if the data had been normalised or not, then we know what has been done

[jacobbieker]

I would lean towards normalizing it when making it a Tensor, just so we have the raw data stored, and if we want to try different normalizations, such as MetNet's which is a bit different, we can easily do that. And normalization shouldn't take too long when loading on the fly

[JackKelly]

This all sounds great!

I agree, maybe nowcasting_dataset shouldn't do any normalisation (which also makes the nowcasting_dataset code easier!). Instead, all data normalisation can be done on-the-fly in nowcasting_dataloader.

[JackKelly]

This also saves us from having to compute means and standard deviations across, potentially, hundreds of TB of raw data. Instaed we can just compute the statistics for the ~1 TB nowcasting_dataset pre-prepared batches (I guess just the training batches?)

[jacobbieker]

Yeah, I think just the training batches, as otherwise it would still leak some data about the validation and test sets

[peterdudfield]

yea good point, perhaps we need a script to run through all the training batches and calculate the mean and or std

[peterdudfield]

Or maybe its ok just to have a rough number for this. Its only normalised for the purpose of the ML model right?

[JackKelly]

I think a script would be good (that runs through the training batches). I think some types of ML models can cope with badly normalised data; but, to enable fair comparison between all types of models, I think we should probably try to normalise as well as we can 🙂

[JackKelly]

I've taken the liberty of changing the title to "Move all data normalisation to nowcasting_dataloader", because it might be good to tidy up all the normalisation :)

[jacobbieker]

There is also normalization in nowcasting-utils here: https://github.com/openclimatefix/nowcasting_utils/blob/main/nowcasting_utils/models/normalization.py incase we want to have all of our normalization code in the same place

[peterdudfield]

There is also normalization in nowcasting-utils here: https://github.com/openclimatefix/nowcasting_utils/blob/main/nowcasting_utils/models/normalization.py incase we want to have all of our normalization code in the same place

This method might only work for one batch? Is it better to normalise over the whole training set

[jacobbieker]

Yeah, it would probably be better to normalize over the whole training set, rather than the batch.

[JackKelly]

I think it's probably easier to reason about the code if the normalisation is done in one place only (e.g. in nowcasting_dataloader). So I'd maybe lean towards removing normalisation from all our other repos, and only doing normalisation in nowcasting_dataloader. What do you think?

[peterdudfield]

yea agree, thats what the two above PRs are doing

[JackKelly]

Cool beans, thank you, sorry I'm a bit behind the curve!

[peterdudfield]

Cool beans, thank you, sorry I'm a bit behind the curve!

all good

[peterdudfield]

Need to move gsp and pv normalization over there too

[JackKelly]

Is all the normalisation code moved from nowcasting_dataset (and data prep scripts) to nowcasting_dataloader, now? (i.e.: can we close this issue now?! 🙂 )

[peterdudfield]

Gsp and PV is not

[peterdudfield]

GSP is probably quite easy to do, but PV might be a longer.

[peterdudfield]

Ill do GSP first

[JackKelly]

SGTM, thanks!