Remove usage of arrow

[VIKTORVAV99]

Description

The built in datetime class is faster and less error prone than arrow so we should migrate to fully using datetime instead of arrow in our parsers.

This probably needs to be done in steps so it's easier to review as it can have a big impact on how parsers are structured.

Parsers:

• parsers/AW.py
• parsers/AX.py
• BG.py - refactor: removed arrow from BG parser #6178
• parsers/BO.py
• parsers/CA_AB.py - Issue 6135 Remove Arrow Usage from CA_AB #6559
• parsers/CA_NS.py
• parsers/CA_SK.py
• parsers/CEB.py
• parsers/CENACE.py
• parsers/CR.py
• parsers/CZ.py
• parsers/DK.py
• parsers/DO.py
• parsers/ES.py
• parsers/ESKOM.py
• parsers/FR-COR_IT-SAR.py
• parsers/FR_O.py
• parsers/GCCIA.py
• parsers/HN.py
• parsers/IE.py
• parsers/IEMOP.py
• parsers/IN_DL.py
• parsers/IN_EA.py
• parsers/IN_HP.py
• parsers/IN_KA.py
• parsers/IN_MH.py
• parsers/KW.py
• parsers/MN.py
• parsers/NO-NO4_SE.py
• parsers/ONS.py
• parsers/PrinceEdwardIsland.py
• parsers/TAIPOWER.py
• parsers/TH.py
• parsers/UA.py
• parsers/US_MISO.py
• parsers/US_NEISO.py
• parsers/VN.py
• parsers/YUKONENERGY.py
• parsers/ajenti.py
• parsers/BD.py
• parsers/CAMMESA.py
• parsers/CA_BC.py - refactor: remove arrow from CA-BC parser #6185
• parsers/CA_NB.py
• parsers/CA_ON.py - refactor: remove arrow from CA-ON parser #6190
• parsers/CA_QC.py - refactor: remove arrow from CA-QC #6192
• parsers/CH.py
• parsers/CL.py
• parsers/CO.py
• parsers/CY.py
• parsers/DK_BHM.py
• parsers/EIA.py
• parsers/ELEXON.py
• parsers/ENTE.py
• parsers/ENTSOE.py
• parsers/ESIOS.py
• parsers/FO.py - refactor: removed arrow from FO parser #6189
• parsers/FR.py
• parsers/GB.py
• parsers/GB_ORK.py
• parsers/GE.py
• parsers/GSO.py
• parsers/GT.py
• parsers/IL.py
• parsers/IN.py
• parsers/IN_PB.py
• parsers/IN_UT.py
• parsers/IN_WE.py
• parsers/IQ.py
• parsers/IS.py
• parsers/JP-KY.py
• parsers/JP.py
• parsers/JP_KN.py
• parsers/KPX.py
• parsers/MD.py
• parsers/NG.py
• parsers/NI.py
• parsers/NL.py
• parsers/NTESMO.py
• parsers/NZ.py
• parsers/OPENNEM.py
• parsers/PA.py
• parsers/PE.py - PE parser: add unit tests and remove usage of arrow module #6481
• parsers/PF.py
• parsers/RU.py
• parsers/SG.py
• parsers/SV.py
• parsers/TR.py
• parsers/US_CA.py
• parsers/US_NY.py
• parsers/US_PJM.py
• parsers/US_PREPA.py
• parsers/US_SPP.py
• parsers/UY.py
• parsers/occtonet.py
• statnett.py

[Barissa-Imran]

Hi @VIKTORVAV99 , I would like to work on this.

[VIKTORVAV99]

Hi @VIKTORVAV99 , I would like to work on this.

Sure thing!

We use arrow in a lot of places so just pick a parser/file and remove it.
It should be replaced by datetime objects and I would prefer one PR per parser as it will allow for much faster reviews.

[Barissa-Imran]

I've compiled the following list to keep track of all parsers that have been refactored and one's the have not. The command I used is:

grep -El '(import arrow|arrow)' parsers/*.py

or the below to get files without arrow:

grep -L -E '(import arrow|arrow)' parsers/*.py

Edit by @VIKTORVAV99:
The list has been moved to the top level issue comment so the tracking works correctly.

[yujia21]

Hello, would like to help too! This is my first time contributing, started with the parser for my home country: #6246

[VIKTORVAV99]

Hello, would like to help too! This is my first time contributing, started with the parser for my home country: #6246

Thats great, thanks!

[gianantoniopini]

Hi, I have just created PR #6481 to remove usage of the arrow module from the PE parser.

[gianantoniopini]

Hallo! I was planning to start trying to remove usage of the arrow module from the CL parser, unless somebody else is already working on it.

[VIKTORVAV99]

Hallo! I was planning to start trying to remove usage of the arrow module from the CL parser, unless somebody else is already working on it.

I don't think anyone is working on that particular parser so go right ahead! And thanks!

[gianantoniopini]

Thanks!

[nboswell216]

I tried to start on CA_BC this evening, but ran into an issue in attempting an initial run test_parser. Here is the full output:
poetry run test_parser CA-BC Traceback (most recent call last): File "<string>", line 1, in <module> File "/workspaces/electricitymaps-contrib/.venv/lib/python3.10/site-packages/click/core.py", line 1134, in __call__ return self.main(*args, **kwargs) File "/workspaces/electricitymaps-contrib/.venv/lib/python3.10/site-packages/click/core.py", line 1059, in main rv = self.invoke(ctx) File "/workspaces/electricitymaps-contrib/.venv/lib/python3.10/site-packages/click/core.py", line 1401, in invoke return ctx.invoke(self.callback, **ctx.params) File "/workspaces/electricitymaps-contrib/.venv/lib/python3.10/site-packages/click/core.py", line 767, in invoke return __callback(*args, **kwargs) File "/workspaces/electricitymaps-contrib/test_parser.py", line 61, in test_parser parser: Callable[..., list[dict[str, Any]] | dict[str, Any]] = PARSER_KEY_TO_DICT[ KeyError: 'CA-BC'

I've tried all permutations of CA-BC with underscores and spaces, and still can't seem to find the key. The web app shows 'CA-BC', and the same zone key is listed multiple times in the parser code.
Any idea on what might be causing this?

[shuuji3]

@nboswell216 When running poetry run test_parser without the second DATA_TYPE option, it looks like it will be the default production. But CA_BC.py doesn't have a fetch_production() function so it failed to run. CA_BC.py instead defines fetch_consumption() so consumption option can be run:

> poetry run test_parser CA-BC consumption
Parser result:
[{'consumption': 7194.0,
  'datetime': datetime.datetime(2024, 2, 20, 5, 0, tzinfo=zoneinfo.ZoneInfo(key='America/Vancouver')),
  'source': 'bchydro.com',
  'sourceType': <EventSourceType.measured: 'measured'>,
  'zoneKey': 'CA-BC'},

[lin0110]

Hi, just wondering if I can still contribute to this issue. I'm first time trying to contribute, very beginner level. Do I just need to change parser to date time object? Would it be possible to see any before fix version and after fix version?

[VIKTORVAV99]

Hi, just wondering if I can still contribute to this issue. I'm first time trying to contribute, very beginner level. Do I just need to change parser to date time object? Would it be possible to see any before fix version and after fix version?

There are still a few parsers that need updating I think but if you want a before and after fix I would look at some of the merged PRs. There you should be able to see both the code before and after as well as the diff between the two.

[lin0110]

Hi, just wondering if I can still contribute to this issue. I'm first time trying to contribute, very beginner level. Do I just need to change parser to date time object? Would it be possible to see any before fix version and after fix version?

There are still a few parsers that need updating I think but if you want a before and after fix I would look at some of the merged PRs. There you should be able to see both the code before and after as well as the diff between the two.

Thank you for clarifying! If no one is working on [parsers/IN.py], can I work on [parsers/IN.py]? Or is there any other suggested parser for beginner? :)

[nboswell216]

Hi, just wondering if I can still contribute to this issue. I'm first time trying to contribute, very beginner level. Do I just need to change parser to date time object? Would it be possible to see any before fix version and after fix version?

There are still a few parsers that need updating I think but if you want a before and after fix I would look at some of the merged PRs. There you should be able to see both the code before and after as well as the diff between the two.

Thank you for clarifying! If no one is working on [parsers/IN.py], can I work on [parsers/IN.py]? Or is there any other suggested parser for beginner? :)

@lin0110 Any parser will be good for a beginner. If you get stuck, check out how the conversion has been handled in other parsers, especially in the same region. No need to ask to work on any of the parsers for this issue - if you don't see any open PR for a parser, just go for it!

[lin0110]

Hi, just wondering if I can still contribute to this issue. I'm first time trying to contribute, very beginner level. Do I just need to change parser to date time object? Would it be possible to see any before fix version and after fix version?

There are still a few parsers that need updating I think but if you want a before and after fix I would look at some of the merged PRs. There you should be able to see both the code before and after as well as the diff between the two.

Thank you for clarifying! If no one is working on [parsers/IN.py], can I work on [parsers/IN.py]? Or is there any other suggested parser for beginner? :)

@lin0110 Any parser will be good for a beginner. If you get stuck, check out how the conversion has been handled in other parsers, especially in the same region. No need to ask to work on any of the parsers for this issue - if you don't see any open PR for a parser, just go for it!

Thank you guys for responding and encouraging me, I really appreciate it! I'm currently working on IN.py, but I'm encountering a ConnectTimeoutError while running 'poetry run test_parser IN'. How can I fix this so I can continue testing my code?