New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Fix parsing of stata dates (#17797) #17990

Merged
merged 5 commits into from Oct 31, 2017

Conversation

Projects
None yet
4 participants
@miker985
Contributor

miker985 commented Oct 26, 2017

  • closes #17797
  • tests added / passed
  • passes git diff upstream/master -u -- "*.py" | flake8 --diff
  • Test and expand parsing of STATA date/time objects.

Expands behavior provided by the following to include most STATA format codes.

        # remove format details from %td
        self.fmtlist = ["%td" if x.startswith("%td") else x
                        for x in self.fmtlist]

Add tests for above behavior (previously untested) + all additional format codes

@codecov

This comment has been minimized.

Show comment
Hide comment
@codecov

codecov bot Oct 26, 2017

Codecov Report

Merging #17990 into master will decrease coverage by 0.01%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #17990      +/-   ##
==========================================
- Coverage   91.23%   91.22%   -0.02%     
==========================================
  Files         163      163              
  Lines       50113    50112       -1     
==========================================
- Hits        45723    45713      -10     
- Misses       4390     4399       +9
Flag Coverage Δ
#multiple 89.03% <100%> (-0.01%) ⬇️
#single 40.31% <0%> (-0.06%) ⬇️
Impacted Files Coverage Δ
pandas/io/stata.py 93.7% <100%> (-0.01%) ⬇️
pandas/io/gbq.py 25% <0%> (-58.34%) ⬇️
pandas/core/frame.py 97.75% <0%> (-0.1%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6779ac0...9ab0681. Read the comment docs.

codecov bot commented Oct 26, 2017

Codecov Report

Merging #17990 into master will decrease coverage by 0.01%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #17990      +/-   ##
==========================================
- Coverage   91.23%   91.22%   -0.02%     
==========================================
  Files         163      163              
  Lines       50113    50112       -1     
==========================================
- Hits        45723    45713      -10     
- Misses       4390     4399       +9
Flag Coverage Δ
#multiple 89.03% <100%> (-0.01%) ⬇️
#single 40.31% <0%> (-0.06%) ⬇️
Impacted Files Coverage Δ
pandas/io/stata.py 93.7% <100%> (-0.01%) ⬇️
pandas/io/gbq.py 25% <0%> (-58.34%) ⬇️
pandas/core/frame.py 97.75% <0%> (-0.1%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6779ac0...9ab0681. Read the comment docs.

@codecov

This comment has been minimized.

Show comment
Hide comment
@codecov

codecov bot Oct 26, 2017

Codecov Report

❗️ No coverage uploaded for pull request base (master@cc7abd9). Click here to learn what that means.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master   #17990   +/-   ##
=========================================
  Coverage          ?   91.25%           
=========================================
  Files             ?      163           
  Lines             ?    50099           
  Branches          ?        0           
=========================================
  Hits              ?    45716           
  Misses            ?     4383           
  Partials          ?        0
Flag Coverage Δ
#multiple 89.06% <100%> (?)
#single 40.24% <0%> (?)
Impacted Files Coverage Δ
pandas/io/stata.py 93.7% <100%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cc7abd9...04ec86f. Read the comment docs.

codecov bot commented Oct 26, 2017

Codecov Report

❗️ No coverage uploaded for pull request base (master@cc7abd9). Click here to learn what that means.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master   #17990   +/-   ##
=========================================
  Coverage          ?   91.25%           
=========================================
  Files             ?      163           
  Lines             ?    50099           
  Branches          ?        0           
=========================================
  Hits              ?    45716           
  Misses            ?     4383           
  Partials          ?        0
Flag Coverage Δ
#multiple 89.06% <100%> (?)
#single 40.24% <0%> (?)
Impacted Files Coverage Δ
pandas/io/stata.py 93.7% <100%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cc7abd9...04ec86f. Read the comment docs.

@gfyoung

This comment has been minimized.

Show comment
Hide comment
@gfyoung

gfyoung Oct 26, 2017

Member

@miker985 : You're going to need to add a whatsnew for this PR. However, as the doc for 0.21.1 is not available yet, just hold tight. Feel free to ping us later if we forget to let you know.

Member

gfyoung commented Oct 26, 2017

@miker985 : You're going to need to add a whatsnew for this PR. However, as the doc for 0.21.1 is not available yet, just hold tight. Feel free to ping us later if we forget to let you know.

@miker985

This comment has been minimized.

Show comment
Hide comment
@miker985

miker985 Oct 26, 2017

Contributor

@gfyoung Is this the comment you're looking for?

I'll wait to hear back on the whatsnew file. If I don't hear back in e.g., a week should I ping you?

Contributor

miker985 commented Oct 26, 2017

@gfyoung Is this the comment you're looking for?

I'll wait to hear back on the whatsnew file. If I don't hear back in e.g., a week should I ping you?

@gfyoung

This comment has been minimized.

Show comment
Hide comment
@gfyoung

gfyoung Oct 26, 2017

Member

If I don't hear back in e.g., a week should I ping you?

@TomAugspurger should be merging the PR for the whatsnew soon, so that works.

Member

gfyoung commented Oct 26, 2017

If I don't hear back in e.g., a week should I ping you?

@TomAugspurger should be merging the PR for the whatsnew soon, so that works.

df = read_stata(self.stata_dates)
unformatted = df.loc[0, column]
formatted = df.loc[0, column + "_fmt"]
assert unformatted == formatted

This comment has been minimized.

@jreback

jreback Oct 27, 2017

Contributor

are these supposed to be datetime64[ns] dtype?

what happens for the ignored formats? should raise?

@jreback

jreback Oct 27, 2017

Contributor

are these supposed to be datetime64[ns] dtype?

what happens for the ignored formats? should raise?

This comment has been minimized.

@miker985

miker985 Oct 27, 2017

Contributor

are these supposed to be datetime64[ns] dtype?

At this point in the code formatted and unformatted are pandas._libs.tslib.Timestamp objects. Every column in df has a dtype of datetime64[ns]

what happens for the ignored formats? should raise?

Ignored formats are not converted to dates (consistent with previous behavior) source

@miker985

miker985 Oct 27, 2017

Contributor

are these supposed to be datetime64[ns] dtype?

At this point in the code formatted and unformatted are pandas._libs.tslib.Timestamp objects. Every column in df has a dtype of datetime64[ns]

what happens for the ignored formats? should raise?

Ignored formats are not converted to dates (consistent with previous behavior) source

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Oct 27, 2017

Contributor

can you add a whatsnew for 0.21.1 (bug fix io section)

Contributor

jreback commented Oct 27, 2017

can you add a whatsnew for 0.21.1 (bug fix io section)

@jreback

minor comment. ping on green.

Show outdated Hide outdated doc/source/whatsnew/v0.21.1.txt

@jreback jreback added this to the 0.21.1 milestone Oct 28, 2017

@jreback jreback merged commit e886af5 into pandas-dev:master Oct 31, 2017

3 checks passed

ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Oct 31, 2017

Contributor

thanks @miker985

Contributor

jreback commented Oct 31, 2017

thanks @miker985

peterpanmj added a commit to peterpanmj/pandas that referenced this pull request Oct 31, 2017

@miker985 miker985 deleted the miker985:improve-stata-date-loading branch Oct 31, 2017

No-Stream added a commit to No-Stream/pandas that referenced this pull request Nov 28, 2017

TomAugspurger added a commit to TomAugspurger/pandas that referenced this pull request Dec 8, 2017

TomAugspurger added a commit that referenced this pull request Dec 11, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment