US MISO Delay

[robweber]

I've looked all over the code and on the MISO website but can't seem to figure out why the MISO data is delayed up to 30 hours on the electricitymap. As you may guess I live in the MISO market so this is really frustrating to me.

I tried refreshing the URL in the parser and it does seem to update every 5-10 min (although at an hour lag) so is this just a function of when you're polling for it? I would imagine within the hour is good enough (ex: when polled at 1:50PM my local time the data in the URL from MISO was from 12:50PM my time). The data on the site is always hours behind. Any info on the inner workings of this would be great. Ultimately I'm trying to use the CO2 signal API in Home Assistant but it's always saying "data unavailable" for my region due to the lag.

[jarek]

Hey,

The MISO zone currently on Electricity Map uses EIA data, which is not realtime.

(You can find this in config/zones.json under US-MIDW-MISO->parsers->production:) https://github.com/tmrowco/electricitymap-contrib/blob/e721e9ed9d4c1abf36cace83b65114ae9f9ea375/config/zones.json#L7124

It looks like the source was changed to EIA in PR #3009 in February 2021 because MISO data was not considered not reliable at the time. From related discussion in issue #2725 it looks like the issue was reporting of pumping up water in a pumped storage hydro plant.

[robweber]

Thank you for the detailed reply. I read through the various comments on the issue in the links you mentioned. For lack of a better way of phrasing it; it's extremely unfortunate that due to no clear solution for the issue a solution was then devised that negatively impacted everyone in that region that would get the benefit of that data. MISO is a fairly large operator in the US, encompassing a lot of people. The delayed data from the EIA parser quite frankly is sort of awful. It makes electricitymap and the underlying API essentially useless for anyone in this region.

According to #1482 this issue is as old as 3 years ago. That particular issue was closed for "no activity" and #2725 was only closed due to the fact that the parser was changed. In all of these threads everyone basically agreed that moving to the delayed parser was a bad idea but yet none of the solutions suggested were considered. Dismissing the "other" category when it is negative seemed to have the least downsides and would allow the real-time data to be used. In this case it won't affect the fuel mix. Granted it doesn't attribute the value correctly to storage but if MISO isn't presenting it that way (either due to their own internal issues or they have some other reason) it makes sense to ignore a non-zero value rather than trying to hack away some workaround to try an interpret the underlying "why" of the negative value.

It would be great if this problem could be revisited as it impacts a lot of people in a large region.

[jarek]

For what it's worth, I personally agree that the error isn't that large and I would personally work around it. But then again I only look at the parsers as a hobby every now and then, I don't deal with trying to reconcile data from a hundred potentially unreliable sources as a job. I commented in #2725 to see if people would be willing to reopen the issue.

[robweber]

Thanks for doing that.

Something to consider is that EIA is gathering this data via MISO as well so one of two things are happening:

• They're doing something with the other category as it isn't showing up
• MISO is reporting it differently via the EIA reporting mechanism

In either case it would be interesting to gather the fuel mix data from MISO directly via the original parser and then compare it to the published EIA data when it becomes available. It could be that doing whatever EIA is already doing could be replicated in a more real time fashion. This information is obviously being integrated or discarded. Electricitymap just has the benefit of not having to deal with it by using the delayed data.

[madsnedergaard]

Thanks for raising the discussion here and for great inputs from you both.

In general we very much agree with you! This is, however, one of the cases where we need to balance between the "free side" and the "business side". In order to do forecasting and provide historical data, the delayed (but consistent) data is more usable - although we really don't like having to make these trade-offs 😞

Instead of looking into switching back to the old parser and validating if that one still performs the same way (which would only affect MISO and leave the other US zones with EIA and delays), we're exploring internally how we can deal with these delayed zones using a more generalised approach.

I can't make any promises to when this will be done, but "fixing" USA on the map is fairly high up on our priorities :)

[robweber]

Thank you, I appreciate the response. It's unfortunate MISO distributes this data in a way that you have to create these trade-offs. I'll continue to watch and see if this changes in the future. FYI I tried to mark an answer for this discussion but GitHub isn't letting me, must be a bug on the UI of the site.