-
-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] perf improvements for strftime #46116
Conversation
… format=None), as suggested by @auderson.
Can you post %timeit results showing the perf improvement? (edit OP) |
Sure. I edited the todo list accordingly. |
… to be able to replace strftime faster. The result can be demonstrated using `fast_strftime`.
First milestone reached: all utility functions required to create the fast datetime formatting strings are there. A Next step: modify |
…and `UnsupportedDatetimeDirective`. Modified `format_array_from_datetime` to use it when `fast_strftime=True` (new arg). Removed `get_datetime_fmt_dct`
…epts `fast_strftime`. Modified `PeriodArray._format_native_types` so that it benefits from `fast_strftime` too.
…ter`, `Datetime64TZFormatter` and `DatetimeIndex`.
…iod_strftime %q format. Also accelerated slightly the _period_strftime.
… code was right, not the docstring. Fixed the docstring
…tting symbols (% or {} depending on formatting style)
…pecial characters. New `Period.fast_strftime` used in `PeriodArray`. Renamed `convert_dtformat` into `convert_strftime_format` and `UnsupportedDatetimeDirective` into `UnsupportedStrFmtDirective`.
Noted - I'll at least try to move the two small fixes for |
…It was not covered by any test! TODO
…l to `is_dates_only` by an assert that the result is False. Added a corresponding test
Quick status on this PR: today I reached a point where all CI checks pass (finally !!!), and therefore I was able to try reproducing the benchmark in #44764 . Results are pretty good !
```python
import time
import numpy as np
import pandas as pd
from joblib import Parallel, delayed
def timer(f): @Timer @Timer @Timer @Timer @Timer index = pd.date_range('2000', '2020', freq='1min') @delayed records = Parallel(10, verbose=10, max_nbytes='500M')( pd.DataFrame(records, columns=['n', 'time_legacy', 'time_manual', import matplotlib.pyplot as plt
|
@smarie thins way too large a PR pls split this up into logical pieces |
@jreback yes I was going to split it as explained above, but I wanted first to reach a "clean" state where CI is happy and the benchmark shows good results. Now I will try (next week) to split this into pieces, but as already mentioned, there is not much I can do here:
|
This is impressive😀 |
…ure/44764_perf_issue
@@ -2340,14 +2392,70 @@ cdef class _Period(PeriodMixin): | |||
object_state = None, self.freq, self.ordinal | |||
return (Period, object_state) | |||
|
|||
def fast_strftime(self, fmt_str: str, loc_s: object) -> str: | |||
"""A faster alternative to `strftime` using string formatting. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nitpick newline after the """
@@ -101,7 +105,8 @@ def format_array_from_datetime( | |||
ndarray[int64_t] values, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if a tslibs.strftime does get implemented, that might be a good home for format_array_from_datetime
…ure/44764_perf_issue � Conflicts: � pandas/_libs/tslibs/period.pyx � pandas/core/arrays/period.py � pandas/tests/io/formats/test_format.py
This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this. |
Hello @smarie, it seems there is a flake8 issue ( |
Thanks @Paul-AUB ! Actually this PR is just here as a reference, it is split in several pieces. #46759 and #46405 are the next bits of this PR to consider, the first is actually quite close to be merged. The second has a side effect in the CI concerning some locale-dependent setting, that needs to be investigated. Once #46405 is merged, I'll finally update this PR or create a new one with the same contents, so that it contains the remaining bit (the engine able to improve any strftime format) |
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen. |
Hi @mroeschke, yes I plan to finish this now that most of its "other aspects" have been merged (through other PRs #46361, #46759, #46405, #47570). I'll try to find some time later this month |
Changelog
Added the default format "%Y-%m-%d %H:%M:%S" as basic format (same as format=None), as suggested by @auderson.
closes PERF: strftime is slow #44764 , closes BUG: Period formatting directive %l (millisecond) and %u (microseconds) provide wrong results #46252, closes BUG:
UnicodeError
when usingPeriod.strftime
with non-utf8 locale #46319Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added an entry in the latest
doc/source/whatsnew/vX.X.X.rst
file if fixing a bug or adding a new feature.Post
timeit
results showing the perf improvements