Fix CAISO timestamp correction

[grgmiller]

CAISO had previously been reporting start-of-hour values to EIA-930, instead of end-of-hour, as instructed, so we had been applying a timestamp correction to shift the values by +1 hours. However, it looks like starting on June 16, 2022, CAISO started correctly reporting end-of-hour values, so we want to end the correction on this date.

Since we have not yet published 2022 OGE data, this should not affect any of our current outputs.

[miloknowles]

Looks good! I would just run this and manually inspect the timestamps (if you haven't already / if that's possible) since there aren't tests for these cleaning functions.

[grgmiller]

So for CAISO specifically, I was able to compare the EIA-930 data with the data that CAISO reports directly on its website. If we look at the below charts, the green line shows the "benchmark" data from CAISO, and the red line shows the cleaned EIA-930 after implementing this time shift. We can see that the data generally aligns before June 16, and then continues to align even better after the 16th (I believe that when CAISO fixed this timestamp issue, they simultaneously fixed an issue where they had been reporting instantaneous MW rather than hourly average MW, which is why the post 6/16 data looks even better.

However, one thing you notice is that CAISO seems to have "fixed" this issue on June 14, then went back to the old method on June 15, then permanently implemented the change on June 16. Maybe this suggests that to be the most accurate, we should end the correction on 6/14, restart it for one day on 6/15, then end it again on 6/16.

Natural gas:

Solar

[grgmiller]

The new code fixes this issue:

It looks like TEPC also fixed their data, but we will need to dig into when and submit a separate PR

[miloknowles]

It would be nice if there was a way to refactor this code into a helper function, and just define the correction_start and correction_end for each region. Maybe in a future PR?

For example, we implemented a common format for specifying timestamp corrections in the EIA scraper: https://github.com/singularity-energy/scrapers/blob/master/scrapers/eia_genfuelmix.py#L87