Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PERF: Series.dropna with non-nan dtype blocks #11159

Merged
merged 1 commit into from
Oct 18, 2015

Conversation

sinhrks
Copy link
Member

@sinhrks sinhrks commented Sep 21, 2015

Minor improvement when dropna actually does nothing. If OK for 0.17, I'll add a release note.

import numpy as np
import pandas as pd

s = pd.Series(np.random.randint(0, 1000, 50000000))

# before
%timeit s.dropna()
#1 loops, best of 3: 1.58 s per loop

# after
%timeit s.dropna()
#1 loops, best of 3: 568 ms per loop

@sinhrks sinhrks added Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Performance Memory or execution speed performance labels Sep 21, 2015
@sinhrks sinhrks added this to the 0.17.1 milestone Sep 21, 2015
self.s = Series(np.random.randint(1, 10, 1000000))

def series_dropna_int64(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the function that actually get times should start with 'time'

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also this dropna only has effect for each time setup is run, which is not every iteration
And set seed (not sure)?

@jreback
Copy link
Contributor

jreback commented Oct 5, 2015

sure, pls add a note in 0.17.1 whatsnew

@jreback
Copy link
Contributor

jreback commented Oct 9, 2015

@sinhrks looks good, can you rebase / add a release note

@sinhrks
Copy link
Member Author

sinhrks commented Oct 15, 2015

Rebased, and added release note. Followings are asv results added in this PR.

    before     after       ratio
  [a89b96d4] [b2cb56a8]
    10.76ms    10.60ms      0.99  series_methods.series_dropna_datetime.time_series_dropna_datetime
-   12.07ms   726.61μs      0.06  series_methods.series_dropna_int64.time_series_dropna_int64

jreback added a commit that referenced this pull request Oct 18, 2015
PERF: Series.dropna with non-nan dtype blocks
@jreback jreback merged commit 53a0db1 into pandas-dev:master Oct 18, 2015
@jreback
Copy link
Contributor

jreback commented Oct 18, 2015

thanks!

@sinhrks sinhrks deleted the dropna_perf branch November 7, 2015 03:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Performance Memory or execution speed performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants