Nova Scotia update parser for data sources and inconsistencies

[jarek]

We're logging error messages like "discarding datapoint at 2021-07-04 18:00:00+00:00 due to coal percentage out of bounds: 0.24559999999999998"

Our current parser rejects points that have less than 25% coal in the fuel mix, in an effort to duplicate validation functionality of the source website (added in #1193). However the source website appears to have changed and AFAICT there is no longer validation there (ChartTodaysPower function in https://www.nspower.ca/bundles/mainjs?v=6UoQDTSr7sd0yMo3fE3LJgJn4iAuAmXV0JumYpeHxnc1). We should at very least relax the validation bounds a little bit, less than 25% coal is now apparently not uncommon. We should also check the known-max-capacities in the validation section of our parser to make sure they're still accurate.

Also, the current/new source website uses a different API endpoint - we use https://www.nspower.ca/library/CurrentLoad/CurrentMix.json, the website now uses https://www.nspower.ca/api/todayspower/powermix. However the new one appears to be inferior as it doesn't have short-term-historical data as the old one does, and as long as the old URL still works it's probably better to keep using it?

We would also want to update source page in any documentation ("real time data sources" list) to https://www.nspower.ca/cleanandgreen/todays-energy-stats

[skovhus]

Thanks for creating this issue @jarek

[jarek]

The source page https://www.nspower.ca/cleanandgreen/todays-energy-stats now returns an empty 404 response, and there's a leftover "Today's Energy Stats" box on pages like https://www.nspower.ca/cleanandgreen/renewable-energy-sources/tidal-power but it doesn't link to anything unlike the boxes next to it :/ It worked as recently as last August: #3284 (comment). Has too much transparency become inconvenient?

The source JSON links linked above still work. But they log too much hydro: currently (Sun 16 Jan 2022, 9pm NS time) they log 1892.58 MW load x 25.34% hydro supply = 479.58 MW (and the hour before it was 554 MW) whereas I'm not finding any sources for new hydro plants that would bring hydro capacity above the previously published 400-420 MW. The error logs at https://storage.googleapis.com/electricitymap-parser-logs/CA-NS.html recently log hydro being too high as the reason for rejecting datapoints.

Imports seem quite low, and there's no break-out for imports from New Brunswick vs Newfoundland. I'm wondering if the Maritime Link transfer from Newfoundland is getting classified as hydro? (From http://www.pub.nl.ca/applications/IslandInterconnectedSystem/DemandStatusReports.htm it looks like in mid-January 2022, Maritime Link was a pretty constant 200 MW export from NL to NS).

Classifying Maritime Link as hydro might be defensible from a "certificate of origin" point of view (Lower Churchill/Maritime Link was meant partially to supply Nova Scotia and rest of mainland with hydro) but not necessarily from actual generation intensity point of view since Newfoundland still has thermal plants.

There's also no available break-down of Newfoundland generation by fuel or its exchanges.

Overall I'm not too pleased with Canadian data openness :/. I could update the parser to remove the outdated validation, but I don't know what to do about the hydro being too high. With electricity policies basically controlled by the provinces, is it time to start writing to politicians?

[jarek]

Right now Nova Scotia is logging 526.43 MW of hydro which far exceeds the capacity of 403 MW per Wikipedia and NS Power. So I'm guessing they're now including imports from Newfoundland in the hydro figure 🤔 .

[jarek]

I dug through the website again today. "Today's energy stats" is still not available on the general-public-facing website. I did find a "daily report" page in the OASIS section https://www.nspower.ca/oasis/system-reports-messages/daily-report which embeds https://resourcesprd-nspower.aws.silvertech.net/oasis/current_report.shtml which has:

• net load
• import via Maritime Link
• wind generation
• and a bunch of other transmission system data

that appear to be updated every 5 minutes. Might be worth digging into it and updating parser to use this information.

Screenshot:

[VIKTORVAV99]

Do we know the mix of Newfoundland?
If it's only hydro, we might be able to do some parser math to deduct it's imports from the hydro generation on Nova Scotia. (Either way we should build a parser for the exchange.)

[jarek]

Nah, there's some thermal as well. https://en.wikipedia.org/wiki/List_of_generating_stations_in_Newfoundland_and_Labrador including https://en.wikipedia.org/wiki/Holyrood_Thermal_Generating_Station 490 MW capacity of oil.

Per https://www.cer-rec.gc.ca/en/data-analysis/energy-markets/provincial-territorial-energy-profiles/provincial-territorial-energy-profiles-newfoundland-labrador.html, Newfoundland and Labrador as a province overwhelmingly generate hydro, but keep in mind much of this is Churchill Falls which is only connected to the Quebec grid. For a long time there wasn't much connection between Labrador and Newfoundland and a project they've been doing to build a link has ran into problems. From the CER page: "The 490 MW oil-fired Holyrood Thermal Generating Station currently generates between 15% and 25% of the island of Newfoundland’s electricity needs annually. Holyrood was once scheduled for shutdown in 2021 after the completion of Muskrat Falls but delays to the project have extended Holyrood’s operating life."

[VIKTORVAV99]

I see so that's not really an option then.

The only other option would be to reach out to them to see if they have some API that split it out that we can access. Do we know if anyone has been in contact with them before?

[Hurc2000]

I am new here, not really a programmer, but an energy professional. Jarek pointed me here. I can provide support, describe how I think Nova Scotia may be adjusted for improved representation. I also have working knowledge of Newfoundland and Labrador to assist with those setups. The group is already aware of the the available data points but I will describe how to use them. I hope to do this for you today.

[Hurc2000]

As already speculated here, Nova Scotia adds the ML delivery to their hydro total. Why? It's primary delivery source is the contracted obligation from its share of Muskrat Falls. Therefore, any delivery to Nova Scotia may be deemed 100% hydro electric sourced, for the sake of argument, 100% of the time. Any additional deliveries would come from Newfoundland Islands hydro assets since their thermal generation would be too high priced for Nova Scotia except in an emergency. So, how best to display this.

My personal preference, which would also be in line with other representations on Electricity Maps, would be to add CA-NL as a tieline. Parse it's value from the Oasis info Jarek identified above, and also subtract it from the NS hydro total so as to not double count. This would also display the data much like the viewer would expect, hydro being NS's internal hydro total.

[Hurc2000]

[Hurc2000]

The only caveat is that the OASIS data is 5-minute, not hourly, but being a set schedule HVDC link, it doesn't move around except for schedule changes.

In addition to what was pointed out above for useful values, the NS Export value is the NB tieline flow which you already parse from NB info.

The NS NL contacts may be reviewed at https://www.emeranl.com/maritime-link/commercial-agreements/phase-i-lower-churchill-project-agreements

[Hurc2000]

I will talk about Newfoundland & Labrador in #2541.

[VIKTORVAV99]

As already speculated here, Nova Scotia adds the ML delivery to their hydro total. Why? It's primary delivery source is the contracted obligation from its share of Muskrat Falls. Therefore, any delivery to Nova Scotia may be deemed 100% hydro electric sourced, for the sake of argument, 100% of the time. Any additional deliveries would come from Newfoundland Islands hydro assets since their thermal generation would be too high priced for Nova Scotia except in an emergency. So, how best to display this.

This is unfortunately not something we can do as it would go against our methodology which we base the carbon footprints on. Since electricity isn't something you can split up once it has been put into the grid we can't assume it's 100% hydro and we can't deduct it from Newfoundland as it would increase the their carbon footprint in an "unfair" way compared to other zones. I'd recommned you check out our blog post about this, it's located here: https://www.electricitymaps.com/blog/flow-tracing

The only caveat is that the OASIS data is 5-minute, not hourly, but being a set schedule HVDC link, it doesn't move around except for schedule changes.

This is not an issue, the 5 minute data will be aggregated to hourly data by the backend.

The NS NL contacts may be reviewed at https://www.emeranl.com/maritime-link/commercial-agreements/phase-i-lower-churchill-project-agreements

We currently don't account for contracts due to the issue I mentioned above.

I will try to locate the proper place to talk about Newfoundland & Labrador, plus pull some info together for those.

I'd suggest opening a discussion for it as they are more suited for these kind of well, discussions.

[Hurc2000]

Thanks for your feedback, Viktor. I am by no means in a position to specify how to do it, merely trying to provide useful input. However, as I stated elsewhere, I'd like to see it presented in the Electricity Maps norms.

As a preview of the Newfoundland info, you can see in the graphic that they schedule the energy as I indicated.