-
Notifications
You must be signed in to change notification settings - Fork 947
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Data for the Philippines #679
Comments
I'm still digging through all the data and trying to get reliable real-time info for everything. Data so far: Major data sources
Regions and interchangesPhilippines electricity is split into three grids, roughly corresponding to Luzon, Visayas, and Mindanao (the three major island groups). These are not official "regions" or "provinces" and don't have their own ISO 3166-2 codes. I would probably suggest making codes up, e.g. PH-L, PH-V, and PH-M, to avoid conflicting with existing ISO 3166-2 codes. I looked into getting GeoJSON for them and it looks feasible from the Natural Earth dataset we use, with small changes to the shell script, I have it figured out. There is currently no interchange between Visayas and Mindanao, see e.g. https://sg.news.yahoo.com/interconnecting-power-supply-mindanao-visayas-132845854.html There is a connection between Luzon and Visayas, but its use is poorly reported (and not real-time). See schedules in https://www.ngcp.ph/weekly-outlook-vis.asp?location=LEYTE_SAMAR and reports in previous-day summary (PDS). On 2017-09-01, next-day summary (NDS) indicated that on 2017-09-02 (Saturday), Visayas would import a peak of 333 MW from Luzon on top of its own 1521 MW local generation, so a non-trivial amount. However, when checked on 2017-09-03, the PDS for Visayas for 2017-09-02 indicated HVDC of 0, with Local Generation being equal to Energy Demand = ex-post peak 1719 MW at 14:00. Similarly PDS for 2017-09-02 for Luzon indicated HVDC of 0 with Local Generation (ex-post peak 8383 MW at 14:00) also being equal to Energy Demand. (Note that you can choose past dates for Previous-Day Summary and Next-Day Summary views.) http://wesm.ph/inner.php/downloads/hvdc_schedule claims to contain past interchange numbers, however I have not seen it return anything other than 0 MW for the few recent dates I've tried, both ex-ante and ex-post. MindanaoThis is the third major island group. There is currently no real-time or historical data for the region on WESM website, though there is some non-real-time data on NGCP website. They are currently being added to WESM, see progress reports at http://wesm.ph/inner.php/downloads/wesm_mindanao_top_weekly_report. Hopefully they'll be added to the other WESM data once complete. DemandDaily demand peaks are on the order of 8400 MW for Luzon and 1850 MW for Visayas. Demand figures on the front page http://wesm.ph/ are loaded from http://wesm.ph/system_lwap_graphs.php from the *DemandVal variables - fairly easy to parse. These figures appear fairly reliable and accurate. Forecasted demand: Luzon NDS, Visayas NDS http://wesm.ph/inner.php/downloads/significant_variations_report claims forecast performance for Visayas is within 5% for 2017-08-25, for Luzon within 3%. For further verification, see e.g.: Per https://www.facebook.com/NGCPph/posts/1533064120083426, on 2017-08-25 Visayas forecast peak demand around 18:00-21:00 was 1775 MW, available capacity 1746 MW. Scraping http://wesm.ph/system_lwap_graphs.php on that day gave actual peak demand 1737 MW at 14:00, between 18 and 21 the peak was 1676 MW at 19:00 (but possibly the announcement lowered the demand). http://wesm.ph/inner.php/downloads/market_prices_&_schedules for Real Time Ex-Post, Generation, Visayas for 2017-08-25 gave 1733 MW at 14:00, and 1722 MW at 19:00. Visayas PDS set for 2017-08-25 gave RTD_DEMAND of 1737 and RTX_DEMAND of 1703 MW at 14:00 (RTD being ex-ante, RTX being ex-post); RTD_DEMAND of 1676 and RTX_DEMAND of 1652 MW at 19:00. (Get the CSV export for a slightly better view of the data). The data seems wonky though, or I don't fully understand it. Values in NDS and PDS don't equal, and the value shown on home page (see Fuel Mix below) is different still. It looks like OFFER in NDS is incomplete, or more companies offer once it's clear that more power will be needed (or once the price goes up). I also have no idea what is going on with the HVDC interchange. Production and fuel mixThis is trickier, particularly for Visayas. The only reliable real-time fuel assignment I've found so far is on the http://wesm.ph/ home page. All other sources are either forecasts or past-day updates that only update once a day after midnight. There are three part-pages that can be loaded in with AJAX: However, the fuel pie chart data is the same in "system" and "luzon" files. It appears that that is the Luzon generation, and Visayas is in the correctly-named file. Past fuel mix for the same day only can be retrieved by appending ?hour= URL param. However, the real-time info reported on the home page doesn't line up. The sum of generation is different than the demand figure shown on the same page. In case of Luzon, generation is 2-4% lower than reported demand. However, in case of Visayas, generation is about 22-28% lower! Timestamps are also a little confusing. Throughout the WESM site, zero-indexed hours and 1-indexed hours (with last datapoint of the day at 23 or 24, respectively) seem to be mixed fairly freely, and confusingly. Requesting http://wesm.ph/visayas_fuel_participants.php?hour=1 gives a file with
Requesting ?hour=0 gives
Fuel mix data updates around 20-30 minutes after the hour, so that at around 02:20 we get data with "0100" and currHour = 2. Past: http://wesm.ph/inner.php/downloads/market_prices_&_schedules has data for previous day only. E.g. at 2017-09-03 22:04 Philippine time it had data up to end of day 2017-09-02. On 2017-09-02 02:22 Philippine time it had data up to end of 2017-09-01. Data consists of production in MW by plant ID. Forecast: http://wesm.ph/inner.php/downloads/day-ahead_projection_generation has forecast by plant ID. Plant ID looks fairly feasible to match to fuel type: Market Network Model reports may be used for mapping plant IDs like 1MARVEL_G02 to more descriptive names and fuel types. http://www.kuryente.org.ph/power-plants also has a list of plants by name and fuel type, but doesn't appear to be exhaustive. In a quick attempt, on Luzon about 95% of projected production (7444 of 7781 MW) for 2017-09-02 22:00 was done by 24 plants, and I was able to fairly easily map 22 of them using the latest Market Network Model document and a search engine. I will try to do a mapping of the plants to fuel type to see if it makes the reported past values line up with either values reported in fuel_participants.php or total values reported in PDS. But if they don't, I'm a bit unsure how to proceed. Simplest would be to take the pie graphs on home page (from fuel_participants.php) at face value and assume no interchange. A little bit better would be to calculate difference between generation and demand and add that in as "unknown". Will that be enough for our purposes? Pragmatically: Luzon is a lot bigger than Visayas, and its data seems better. So maybe the 28% "unknown" for Visayas wouldn't matter so much. |
@jarek Being serious this is great work. I think with all the uncertainties involved in trying to extrapolate from the other data sources it's better to stick with simply taking the generation from the pie charts and allowing for some unknown amount like you say. With Mindanao hopefully being added to the site soon we may find that the data set is improved substantially re: interconnections/reliability. Rather than trying to build a more complex parser I think a fairly simple one will better for adapting to changes in the near future. If you want I can help out with the Plant ID mapping? |
Sure :) If you want to play detective, having these plants mapped will be useful for later reference e.g. comparing reported fuel mix with reported production by plant. These are sorted in descending production order based on production last week (based on CSV exported from http://wesm.ph/inner.php/downloads/market_prices_&_schedules). The first ~20 will get us most of the way there, the last ~20 are usually zero production. Plants with same prefix ('3ILIJAN_G02' and '3ILIJAN_G01') are usually the same type, but not always, e.g. 1BURGOS_G01 is wind while 1BURGOS_G02 and _G03 are solar. Luzon: ['1SUAL_G01', # coal Visayas: ['4LEYTE_A', |
@jarek |
Thanks, that's fantastic. I will try to put a parser-and-comparison-engine together. Will report back. |
So, this is still on my todo list, but unfortunately I don't yet anything ready to share. I have been collecting live-reported data on and off, but have not processed it or compared to post-reported data yet. |
@jarek here is some more support :) on https://www.doe.gov.ph/list-existing-power-plants you'll find a list with the installed capacities and also a list with the operating power plants (in case anything is not yet mapped) as of June 2017: LUZON:
VISAYAS:
MINDANAO:
PHILIPPINES TOTAL:
|
Hello, I dug into this a little bit and unfortunately have no good news. At least for Visayas on 2018-03-02, data reported in real-time in the pie charts on WESM front page was substantially different from data reported afterwards in WESM's "downloads"/reporting section. Sources of information
For interested, data in CSV: https://gist.github.com/jarek/e8b2c15321da87f754530c39921badbf - data I collected is near "scraped: Vis demand" and "generation total", and then "chart biomass", "chart coal", etc The graphs go up to 21st hour because I didn't get later "chart" data (also because data for 23rd hour is very hard to get because of how the data rolls over). Biggest problems
Supply vs demand
Reported generation vs MPS RTX and RTDCoal and geothermal, the two biggest fuels, differ significantly in what was reported on the hour vs what was reported later. Solar is good and oil tracks fairly well as well. Here is how much coal and geothermal are missing. It's unfortunately not static over the day, either... Bad news summaryI can't find a way to make sense of the Visayas 2018-03-02 data. I will try Luzon and maybe another day to see if that looks better. If not, I might try emailing them asking what's up... |
Thanks for investigating @jarek. Maybe the mismatch will be fixed in the future. |
I created the "historical data" label for zones with available generation data for the past. 🏷 Isn't it possible to use the real-time mix and fill the gap between the generation per type and the demand with "unknown" (like for Kyushu/Japan and for NL)? In a second step you could fill up or overwrite it with the ex-post data of the per-unit generation. Is this an option? |
I have to admit I haven't checked on this in a while and I never contacted WESM about the inconsistencies. I will try to look again and see if the data matches up now. |
Per http://wesm.ph/inner.php/news/news_and_events/574, WESM and Philippine Electricity Market Corporation might have created or activated a new body called Independent Electricity Market Operator of the Philippines with website http://www.iemop.ph/ which might possibly have with more or different data. They seem to have generation mix for Mindanao now, for one thing. I have not yet checked if the values sum up to the reported demand, or match post-reported values. |
The data on WESM.ph seems to be no longer available, but IEMOP.ph looks very promising. In fact, it looks like IEMOP is the old WESM site; you can replace http://wesm.ph with http://iemop.ph to get the links working again, though the data is no longer available http://www.iemop.ph/ also has hourly demand per region at the top of the page -
so that's really old and not useful. However, via here I found http://180.232.127.40/views/IEMOP_SITE_SUMM1/STORY1?iframeSizedToWindow=true&:embed=y&:showAppBanner=false&:display_count=no&:showVizHome=no, which has hourly supply/demand data (where supply means 'available capacity'), as well as generation mix (button at the top of the page). This page seems to be updated hourly as well. (I checked at two different times on the same day and the values had changed in the mean time.) Note: given that the generation mix page reports 2.5% solar despite it being night there, I think this is the daily generation mix so far - but if electricitymap does fetch this data each hour, because demand data is also available, it should be possible to do the math to figure out the generation mix per hour. |
@q-- at this point I expect you to find real-time generation data for China in the coming days 😂Amazing detective work! |
Any chance that this data is better now? It looks like there might be some potential real time data now... https://www.iemop.ph/the-market/market-dashboards/real-time-outcomes/ |
Thanks for all the valuable information shared here! 🙌 We're closing this issue and adding to our main issue for missing countries as it doesn't currently seem possible to build a real-time parser given the data provided here. |
It does, check the latest link provided by @gwpicard. Generation breakdown by type is still by day, but the information is enough to build an Israel- or Kuwait type parser. Also, I just found a few more useful things:
|
I'll reopen! In this case, we just need a good indicator for the breakdown. We could use the weekly summary reports as a baseline perhaps? |
Yes; besides the hard-to-parse graph, the weekly reports also have average production by generation type by day. However, if you're looking to calculate the average fuel mix over a 12-month period, the Market Operations Highlights are probably better, as they have the fuel mix per month. As an aside, I also found precise generation by plant code, similar to what @jarek found earlier before most content was removed from the old WESM website, one CSV per day, here, updated until yesterday. Mapping plant codes to generation type is going to be challenging though. (Plant codes also seem to be subtly different from those found by @jarek, or there are just a lot of new plants.) A lot of other information, such as pricing, generation forecasts etc is also available as CSV via the markets data page. |
https://www.iemop.ph/market-data/rtd-prices-and-schedules/ might be the ticket: Throw that in pivot tables, you get: Lookup one of the resources from: IE: 04LEYTE_A -> 340 MW scheduled for Registered capacity: WESM participant: Not trivial to map those to specifc types, but registered capacity lists: 390 entries for a given day; many of them individual units in the same power station. |
It's been a while since I looked into this issue, but if I'm not mistaken the format has changed. There used to be a generator code column somewhat similar to this, and like those in your second comment. |
I am closing this issue now as we have a parser for Philippines that was added in #5769 and suggest opening a new issue for any questions around the parser or data used in the parser if they arise. |
http://www.wesm.ph/ has hourly generation breakdown for Luzon and Visayas. These are two of Philippines' three major island groups, covering about 75% of the population.
According to news reports, the third island group, Mindanao, was to join or has joined WESM in June or July 2017, but there seems to be no data yet. Maybe they're still integrating it.
I would like to work on this next week :)
The text was updated successfully, but these errors were encountered: