/pandas

PERF: Improve replace perf #12745

Closed
wants to merge 1 commit into
from

Conversation

Reviewers
No reviews
Assignees
No one assigned
Labels
Dtypes Performance
Projects
None yet
Milestone
  0.19.2
3 participants
@sinhrks @jreback @codecov-io
@sinhrks
Member

sinhrks commented Mar 30, 2016

  • tests added / passed
  • passes git diff upstream/master | flake8 --diff
  • whatsnew entry

When .replace is called with dict, replacements are done per value. Current impl try to soft convert the dtype in every replacement, but it is enough to be done in the final replacement.

Bench

-  712.83ms   355.93ms      0.50  replace.replace_convert.time_replace_series_timestamp
-     1.50s   698.21ms      0.46  replace.replace_convert.time_replace_frame_timestamp
-     3.12s   690.48ms      0.22  replace.replace_convert.time_replace_frame_timedelta
-     1.69s   354.83ms      0.21  replace.replace_convert.time_replace_series_timedelta

sinhrks added the Performance label Mar 30, 2016

sinhrks added this to the 0.18.1 milestone Mar 30, 2016

@jreback
Contributor

jreback commented Mar 30, 2016

notice that BoolBlock.replace doesn't have convert arg at all! so not tested / working. can you give a check.
@sinhrks
Member

sinhrks commented Mar 30, 2016

Ah, found some .replace() conversion looks buggy. I'll submit a separate issue as the fix not looks straightforward.

@jreback jreback commented on an outdated diff Mar 30, 2016

pandas/tests/series/test_replace.py
@@ -0,0 +1,276 @@
+# coding=utf-8
+# pylint: disable-msg=E1101,W0612
@jreback

jreback Mar 30, 2016

Contributor

great!
@sinhrks
Member

sinhrks commented Mar 30, 2016

See #12747. Change this PR title to WIP until #12747 is fixed.

sinhrks changed the title from PERF: Improve replace perf to WIP: PERF: Improve replace perf Mar 30, 2016

jreback removed this from the 0.18.1 milestone Apr 17, 2016

@codecov-io

codecov-io commented May 21, 2016 • edited

Current coverage is 85.27% (diff: 90.00%)

Merging #12745 into master will decrease coverage by <.01%

@@             master     #12745   diff @@
==========================================
  Files           144        144          
  Lines         50915      50921     +6   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43419      43423     +4   
- Misses         7496       7498     +2   
  Partials          0          0

Powered by Codecov. Last update dfeae39...ffc59b0
@jreback
Contributor

jreback commented Sep 9, 2016

can you rebase / update?

jreback added the Dtypes label Nov 23, 2016

@jreback
Contributor

jreback commented Nov 23, 2016

lgtm. can we merge this independent of the other PR's? (so it could go in 0.19.2)
@sinhrks sinhrks PERF: Improve replace perf

All checks have passed

2 successful checks
continuous-integration/appveyor/pr — AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr — The Travis CI build passed
Details
ffc59b0

sinhrks changed the title from WIP: PERF: Improve replace perf to PERF: Improve replace perf Nov 29, 2016

@sinhrks
Member

sinhrks commented Nov 29, 2016 • edited

It is independent. #12780 fixes some existing bugs separately.

jreback added this to the 0.19.2 milestone Nov 30, 2016

jreback closed this in e299560 Nov 30, 2016

@jreback
Contributor

jreback commented Nov 30, 2016

thanks!

sinhrks deleted the sinhrks:replace_perf branch Nov 30, 2016

@jorisvandenbossche jorisvandenbossche added a commit that referenced this pull request Dec 15, 2016

@sinhrks @jorisvandenbossche sinhrks + jorisvandenbossche [Backport #12745] PERF: Improve replace perf 
When .replace is called with
`dict`, replacements are done per value. Current impl try to soft
convert the dtype in every replacement, but it is enough to be done in
the final replacement.

Author: sinhrks <sinhrks@gmail.com>

Closes #12745 from sinhrks/replace_perf and squashes the following commits:

ffc59b0 [sinhrks] PERF: Improve replace perf

(cherry picked from commit e299560)
 7479d41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment