PH production & exchanges parser

[pierresegonne]

Issue

We don't have data for the Philippines, yet it is available

Description

• Adds production parser
• Adds exchange parser
• Adds exchanges config

Preview

Double check

• I have tested my parser changes locally with poetry run test_parser "zone_key"
• I have run pnpx prettier --write . and poetry run format to format my changes.

[pierresegonne]

We need to remove that, I forgot

[jarek]

Is this based on #679?

[VIKTORVAV99]

Overall it work well with a few fixes (that we need to do). I have added more details in individual comments.

I think I'll pause the review here and do another one once the needed fixes are implemented at least.

PS: I think this parser could benefit from multithreading while fetching all the individual files.

PPS: I added the parser to the zone configs.

[pierresegonne]

Is this based on #679?

We did use some of the info dug at that time as a base :)
We could add you guys in the contributors for this

[pierresegonne]

Thanks @VIKTORVAV99 for the thorough review, sorry if it was still a bit messy

[pierresegonne]

@VIKTORVAV99 Ready to go, I did not implement multi-threading for parsing the production csvs because I think we're quite far from hitting the timeout, let's do it in a subsequent PR if we see that it happens?

[VIKTORVAV99]

There are some type erros still but nothing blocking as far as I am concerned. They seem to stem from the named tuples and dataframe changing the types (but not content).

Just one final note about the source (and one type error) but I'm approving already.

Nice work on this

[VIKTORVAV99]

Just one final as I just saw this in the console (should have scrolled up earlier 🙄):

2023-08-24 14:08:58,443 WARNING  test_parser                    PH - production: Unknown resources found: ['1AMBUK_SS' '1AMBUK_T1L1' '1ANGAT_A' '1ANGAT_M' '1ANGAT_SS' '1ANGAT_T5L1'
 '1BAKUN_SS' '1BALNEO_SS' '1BALSIK_SS' '1BANTAY_SS' '1BATAAN_SS'
 '1BAUANG_GS1' '1BAUANG_GS2' '1BAUANG_GS3' '1BAUANG_SS' '1BAUAN_T1L2'
 '1BCNOT_SS' '1BINGA_SS' '1BNTAY_T1L3' '1BOTOLAN_SS' '1BPPC_SS'
 '1BTPP_T3L1' '1BTPP_T3L3' '1BTPP_T3L5' '1BTPP_T4L1' '1BTSOLEN_BAT'
 '1BURGOS_SS' '1BYOMB_SS' '1CASECN_SS' '1CBNTU_SS' '1CBNTU_T1L3'
 '1CLARK_L01' '1CLARK_L02' '1CLARK_L03' '1CLARK_L04' '1CLARK_SS'
 '1CLARK_T1L1' '1CLARK_T1L2' '1CNCEP_BAT' '1CNCEP_SS' '1CRUZD_SS'
 '1CURIM_SS' '1DUHAT_SS' '1EHVSJ_SS' '1EHVSM_SS' '1GAMU_BATL' '1GAMU_SS'
 '1GNPD_SS' '1HANJI_SS' '1HANJI_T2L1' '1HERMO_FBL' '1HERMO_SS'
 '1IPOWER_G02' '1KDAM_SS' '1LAMAO_BAT' '1LAMAO_SS' '1LAOAG_SS'
 '1LAOAG_T1L1' '1LATRI_SS' '1LBRDR_SS' '1LIMAY_BAT' '1LIMAY_SS'
 '1MAEC_S01' '1MAGAPIT_BAT' '1MAGAT_BAT' '1MAGAT_SS' '1MALOL_SS'
 '1MARVEL_AUX' '1MARVEL_SS' '1MASIWA_SS' '1MEXICO_SS' '1MEX_AEC'
 '1MPGC_SS' '1MSINLO_BAT' '1NSNMN_L01' '1NSNMN_SS' '1NSNMN_T1L1'
 '1NWIND_L01' '1OLONG_SS' '1PAGUD_SS' '1PETRON_G01' '1PNTBNG_SS' '1SMC_SS'
 '1SNEST_SS' '1SNJOSE_SS' '1SNJOS_T1L3' '1SNMAN_SS' '1SNMAN_T1L2'
 '1SNM_PNELC3' '1SNTGO_SS' '1SROQUE_SS' '1SUAL_T3L1' '1SUBIC_T3L1'
 '1S_ENRO_SS' '1TGEGA_SS' '1TRASU_L01' '1T_ASI_T3L1' '3ALMNOS_BAT1'
 '3ALMNOS_BAT2' '3AVION_SS' '3BACMAN_L01' '3BATAN_SS' '3BATAN_T2L2'
 '3BINAN_SS' '3BOTOCA_SS' '3CALAMBA_SS' '3CALAU_SS' '3CALIRY_SS'
 '3CALIRY_SS1' '3DARAG_T1L1' '3DAR_STDOM' '3DASMA_SS' '3GUMAC_SS'
 '3ILIJAN_SS2' '3INGRID_GS1' '3INGRID_GS2' '3INGRID_GS3' '3INGRID_GS4'
 '3INGRID_GS5' '3INGRID_GS6' '3KAL_SS' '3LABO_SS' '3MKBNC_SS' '3MKBND_SS'
 '3MKBNE_SS' '3MKBN_A' '3MKBN_B' '3MKBN_C' '3MKBN_D' '3MKBN_E'
 '3MKBN_SLACE' '3MLAYA_L01' '3MLAYA_SS' '3MLAYA_SS1' '3MLAYA_T2L1'
 '3MLAYA_T2L2' '3NAG_SS' '3PGBIL_SS' '3PGBIL_SS1' '3PGBIL_SS2' '3QPPL_SU1'
 '3QPPL_T5L1' '3SBPL_L01' '3SLPGC_SS' '3SLTEC_SS' '3SNGAB_S01' '3STAR_SS1'
 '3STAR_SS2' '3STAR_SS3' '3STAR_SS4' '3STAR_SS5' '3STAR_SS6' '3TAYAB_SS'
 '3TAYAB_T1L1' '3TIWIA_SS' '3TIWIC_L01' '3TIWIC_SS' '3TIWI_A' '3TIWI_C'
 '1OLONG_T3L3' '1BTPP_T3L4' '1MEX_SKK' '1DUHAT_T1L1' '1BNTAY_T1L1'
 '1CLARK_T2L1' '3MKBNA_SS']. Please report this issue to the Electricity Maps team

Is that something we need to address first?

[pierresegonne]

Is that something we need to address first?

Not directly, we'll leave the parser running and collect the non matched resources names in a week or so to try to match the remaining ones. By default, the non-matched will be set as unknown prod

[pierresegonne]

There are some type erros still but nothing blocking as far as I am concerned. They seem to stem from the named tuples and dataframe changing the types (but not content).

I'm curious to know, what command do you run to check this?

[VIKTORVAV99]

There are some type erros still but nothing blocking as far as I am concerned. They seem to stem from the named tuples and dataframe changing the types (but not content).

I'm curious to know, what command do you run to check this?

I'm using the Pylance extension in VSCode that does type checking.
Looks like this:

I think you can get the same results with pyright though as that is what pylance is using under the hood.

[xomanuel]

Thank you for this PH parser, I've been waiting for this since I first learned about electricity maps and started contributing, Although I haven't updated the capacity for PH for a while now.

I would like to inform that not all power plants in the PH are available in the Wholesale Electricity Market of the Philippines (WESM) governed by the Independent Electricity Market Operator of the Philippines (IEMOP). The plants listed under the WESM are those plants that are mandated to register based on the following parameters:

• with Pmax equal to or above the following regional thresholds:
  a. Luzon: 10MW
  b. Visayas: 5MW
  c. Mindanao: 5MW

• with Pmax below regional thresholds that have a bilateral contract outside its host Distribution Utility, or intends to sell to the WESM, or inject power to the Grid

• Feed-in-Tariff eligible RE Plants

The PH has several power plants that are not registered under the WESM such as off-grid power plant operators. Unfortunately, no data is available online with regard to their operations.

Under the Enhanced WESM design, power plant operators are required to submit/nominate their capacity at a 5-minute interval.

---

Zones

The Market Network Model of Enhanced WESM has 14 Zones as of September 2023:

1 - North Luzon
2 - South Luzon
3 - Metro Manila
4 - Leyte/Samar
5 - Cebu
6 - Negros
7 - Bohol
8 - Panay
9 - North Western
10 - Lanao
11 - North Central
12 - North Eastern
13 - South Eastern
14 - South Western

1 - 3 are located Luzon Grid
4 - 8 are located Visayas Grid
9 - 14 are located Mindanao Grid

File naming

Generator Resources/Plant Resources:

01AMBUK_U01

01 denotes that the power plant is located in North Luzon Zone
AMBUK is the facility name
U01 is the generator unit also G01 (Note: G01 may denote multiple units or plant as a whole)

Hence, 01AMBUK_U01 is a power plant connected to the North Luzon Grid

Station Name (Substation):

01AMBKLAO

01 denotes that the power plant is located in North Luzon Zone
AMBKLAO denotes station name (Substation)

Hence, 01AMBKLAO is a substation connected to the North Luzon Grid

Load Resources:

01AMBUK_SS

01 denotes that the power plant is located in North Luzon Zone
SS / T1L1 /BAT1/ etc. denotes as a Load

Hence, 01AMBUK_SS is a load connected to the North Luzon Grid

Just one final as I just saw this in the console (should have scrolled up earlier 🙄):

2023-08-24 14:08:58,443 WARNING  test_parser                    PH - production: Unknown resources found: ['1AMBUK_SS' '1AMBUK_T1L1' '1ANGAT_A' '1ANGAT_M' '1ANGAT_SS' '1ANGAT_T5L1'
 '1BAKUN_SS' '1BALNEO_SS' '1BALSIK_SS' '1BANTAY_SS' '1BATAAN_SS'
 '1BAUANG_GS1' '1BAUANG_GS2' '1BAUANG_GS3' '1BAUANG_SS' '1BAUAN_T1L2'
 '1BCNOT_SS' '1BINGA_SS' '1BNTAY_T1L3' '1BOTOLAN_SS' '1BPPC_SS'
 '1BTPP_T3L1' '1BTPP_T3L3' '1BTPP_T3L5' '1BTPP_T4L1' '1BTSOLEN_BAT'
 '1BURGOS_SS' '1BYOMB_SS' '1CASECN_SS' '1CBNTU_SS' '1CBNTU_T1L3'
 '1CLARK_L01' '1CLARK_L02' '1CLARK_L03' '1CLARK_L04' '1CLARK_SS'
 '1CLARK_T1L1' '1CLARK_T1L2' '1CNCEP_BAT' '1CNCEP_SS' '1CRUZD_SS'
 '1CURIM_SS' '1DUHAT_SS' '1EHVSJ_SS' '1EHVSM_SS' '1GAMU_BATL' '1GAMU_SS'
 '1GNPD_SS' '1HANJI_SS' '1HANJI_T2L1' '1HERMO_FBL' '1HERMO_SS'
 '1IPOWER_G02' '1KDAM_SS' '1LAMAO_BAT' '1LAMAO_SS' '1LAOAG_SS'
 '1LAOAG_T1L1' '1LATRI_SS' '1LBRDR_SS' '1LIMAY_BAT' '1LIMAY_SS'
 '1MAEC_S01' '1MAGAPIT_BAT' '1MAGAT_BAT' '1MAGAT_SS' '1MALOL_SS'
 '1MARVEL_AUX' '1MARVEL_SS' '1MASIWA_SS' '1MEXICO_SS' '1MEX_AEC'
 '1MPGC_SS' '1MSINLO_BAT' '1NSNMN_L01' '1NSNMN_SS' '1NSNMN_T1L1'
 '1NWIND_L01' '1OLONG_SS' '1PAGUD_SS' '1PETRON_G01' '1PNTBNG_SS' '1SMC_SS'
 '1SNEST_SS' '1SNJOSE_SS' '1SNJOS_T1L3' '1SNMAN_SS' '1SNMAN_T1L2'
 '1SNM_PNELC3' '1SNTGO_SS' '1SROQUE_SS' '1SUAL_T3L1' '1SUBIC_T3L1'
 '1S_ENRO_SS' '1TGEGA_SS' '1TRASU_L01' '1T_ASI_T3L1' '3ALMNOS_BAT1'
 '3ALMNOS_BAT2' '3AVION_SS' '3BACMAN_L01' '3BATAN_SS' '3BATAN_T2L2'
 '3BINAN_SS' '3BOTOCA_SS' '3CALAMBA_SS' '3CALAU_SS' '3CALIRY_SS'
 '3CALIRY_SS1' '3DARAG_T1L1' '3DAR_STDOM' '3DASMA_SS' '3GUMAC_SS'
 '3ILIJAN_SS2' '3INGRID_GS1' '3INGRID_GS2' '3INGRID_GS3' '3INGRID_GS4'
 '3INGRID_GS5' '3INGRID_GS6' '3KAL_SS' '3LABO_SS' '3MKBNC_SS' '3MKBND_SS'
 '3MKBNE_SS' '3MKBN_A' '3MKBN_B' '3MKBN_C' '3MKBN_D' '3MKBN_E'
 '3MKBN_SLACE' '3MLAYA_L01' '3MLAYA_SS' '3MLAYA_SS1' '3MLAYA_T2L1'
 '3MLAYA_T2L2' '3NAG_SS' '3PGBIL_SS' '3PGBIL_SS1' '3PGBIL_SS2' '3QPPL_SU1'
 '3QPPL_T5L1' '3SBPL_L01' '3SLPGC_SS' '3SLTEC_SS' '3SNGAB_S01' '3STAR_SS1'
 '3STAR_SS2' '3STAR_SS3' '3STAR_SS4' '3STAR_SS5' '3STAR_SS6' '3TAYAB_SS'
 '3TAYAB_T1L1' '3TIWIA_SS' '3TIWIC_L01' '3TIWIC_SS' '3TIWI_A' '3TIWI_C'
 '1OLONG_T3L3' '1BTPP_T3L4' '1MEX_SKK' '1DUHAT_T1L1' '1BNTAY_T1L1'
 '1CLARK_T2L1' '3MKBNA_SS']. Please report this issue to the Electricity Maps team

Is that something we need to address first?

The above are identified as Load Resources.

Hope this information will help in enhancing the parser for PH.

Thank you all!