PERF: Series.dropna with non-nan dtype blocks #11159

Merged
merged 1 commit into from Oct 18, 2015

Conversation

Projects
None yet
4 participants
Member

sinhrks commented Sep 21, 2015

Minor improvement when dropna actually does nothing. If OK for 0.17, I'll add a release note.

import numpy as np
import pandas as pd

s = pd.Series(np.random.randint(0, 1000, 50000000))

# before
%timeit s.dropna()
#1 loops, best of 3: 1.58 s per loop

# after
%timeit s.dropna()
#1 loops, best of 3: 568 ms per loop

sinhrks added this to the 0.17.1 milestone Sep 21, 2015

@jorisvandenbossche jorisvandenbossche and 1 other commented on an outdated diff Sep 21, 2015

asv_bench/benchmarks/series_methods.py
+ self.s2.nsmallest(3, take_last=False)
+
+
+class series_dropna_int64(object):
+ goal_time = 0.2
+
+ def setup(self):
+ self.s = Series(np.random.randint(1, 10, 1000000))
+
+ def series_dropna_int64(self):
@jorisvandenbossche

jorisvandenbossche Sep 21, 2015

Owner

the function that actually get times should start with 'time'

@kawochen

kawochen Sep 22, 2015

Contributor

Also this dropna only has effect for each time setup is run, which is not every iteration
And set seed (not sure)?

Contributor

jreback commented Oct 5, 2015

sure, pls add a note in 0.17.1 whatsnew

Contributor

jreback commented Oct 9, 2015

@sinhrks looks good, can you rebase / add a release note

Member

sinhrks commented Oct 15, 2015

Rebased, and added release note. Followings are asv results added in this PR.

    before     after       ratio
  [a89b96d4] [b2cb56a8]
    10.76ms    10.60ms      0.99  series_methods.series_dropna_datetime.time_series_dropna_datetime
-   12.07ms   726.61μs      0.06  series_methods.series_dropna_int64.time_series_dropna_int64

@jreback jreback added a commit that referenced this pull request Oct 18, 2015

@jreback jreback Merge pull request #11159 from sinhrks/dropna_perf
PERF: Series.dropna with non-nan dtype blocks
53a0db1

@jreback jreback merged commit 53a0db1 into pandas-dev:master Oct 18, 2015

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Contributor

jreback commented Oct 18, 2015

thanks!

sinhrks deleted the sinhrks:dropna_perf branch Nov 7, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment