-
Notifications
You must be signed in to change notification settings - Fork 906
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor(JP): remove arrow usage #6572
base: master
Are you sure you want to change the base?
Conversation
ce499ee
to
35e1012
Compare
@@ -210,9 +215,14 @@ def fetch_consumption_forecast( | |||
# Currently past dates not implemented for areas with no date in their demand csv files | |||
if target_datetime and zone_key == "JP-HKD": | |||
raise NotImplementedError("Past dates not yet implemented for selected region") | |||
datestamp = arrow.get(target_datetime).to("Asia/Tokyo").strftime("%Y%m%d") | |||
|
|||
if target_datetime is None: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When target_datetime == None
, arrow.get(target_datetime)
returns the current time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unfortunately running poetry run test_parser JP-CB
fails with:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/viktor/Repos/electricitymaps-contrib/.venv/lib/python3.10/site-packages/click/core.py", line 1134, in __call__
return self.main(*args, **kwargs)
File "/Users/viktor/Repos/electricitymaps-contrib/.venv/lib/python3.10/site-packages/click/core.py", line 1059, in main
rv = self.invoke(ctx)
File "/Users/viktor/Repos/electricitymaps-contrib/.venv/lib/python3.10/site-packages/click/core.py", line 1401, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/viktor/Repos/electricitymaps-contrib/.venv/lib/python3.10/site-packages/click/core.py", line 767, in invoke
return __callback(*args, **kwargs)
File "/Users/viktor/Repos/electricitymaps-contrib/test_parser.py", line 66, in test_parser
res = parser(
File "/Users/viktor/Repos/electricitymaps-contrib/parsers/lib/config.py", line 20, in wrapped_f
result = f(*args, **kwargs)
File "/Users/viktor/Repos/electricitymaps-contrib/parsers/JP.py", line 69, in fetch_production
df = fetch_production_df(zone_key, session, target_datetime)
File "/Users/viktor/Repos/electricitymaps-contrib/parsers/JP.py", line 135, in fetch_production_df
df = pd.merge(df, df2, how="inner", on="datetime")
File "/Users/viktor/Repos/electricitymaps-contrib/.venv/lib/python3.10/site-packages/pandas/core/reshape/merge.py", line 110, in merge
op = _MergeOperation(
File "/Users/viktor/Repos/electricitymaps-contrib/.venv/lib/python3.10/site-packages/pandas/core/reshape/merge.py", line 707, in __init__
self._maybe_coerce_merge_keys()
File "/Users/viktor/Repos/electricitymaps-contrib/.venv/lib/python3.10/site-packages/pandas/core/reshape/merge.py", line 1346, in _maybe_coerce_merge_keys
raise ValueError(msg)
ValueError: You are trying to merge on object and datetime64[ns, tzfile('/usr/share/zoneinfo/Asia/Tokyo')] columns. If you wish to proceed you should use pd.concat
This needs to be fixed before this can be merged.
date = row["Date"] | ||
time = row["Time"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it not be easier to parse these directly using the date and time libraries and then just combine them instead of using string manipulation?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks only datetime
library has strptime()
but there is no string parser in date
and time
standard libraries (ref. https://docs.python.org/3.10/library/datetime.html#strftime-strptime-behavior). I wonder if there's another method other than datetime.strptime()
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
datetime.time
has their own parsing functions, same with date.
https://docs.python.org/3.10/library/time.html#time.strptime
You could also use the date time library directly on each of them and just call .date()
or .time()
to get the individual parts.
Issue
#6135
Description
It looks like the price exchange data source was updated and failed to run it by
poetry run ./test_parser.py JP-KY price
so I couldn't test the change forprice
task.Execution result on
master
and this branch:Preview
N/A
Double check
poetry run test_parser "zone_key"
pnpx prettier@2 --write .
andpoetry run format
in the top level directory to format my changes.