Fix to csv2rec bug for review #6185

Merged
merged 5 commits into from Mar 20, 2016

Conversation

Projects
None yet
4 participants
Contributor

ultra-andy commented Mar 18, 2016

Datetime strings in csv files in dayfirst or yearfirst format with
nonzero hour&minute&second will not be properly represented. Date
strings will, presuming the csv file contains no strings in that column
that have nonzero hour/minute/second components in a datetime string. Refer to issue #6184 for further information.

@ultra-andy ultra-andy Fix to csv2rec bug for review
Datetime strings in csv files in dayfirst or yearfirst format with
nonzero hour&minute&second will not be properly represented. Date
strings will, presuming the csv file contains no strings in that column
that have nonzero hour/minute/second components in a datetime string.
36a6d81

mdboom added the needs_review label Mar 18, 2016

@tacaswell tacaswell commented on an outdated diff Mar 19, 2016

lib/matplotlib/mlab.py
@@ -2833,7 +2833,13 @@ def mybool(x):
raise ValueError('invalid bool')
dateparser = dateutil.parser.parse
- mydateparser = with_default_value(dateparser, datetime.date(1, 1, 1))
+
+ def mydateparser(x):
+ # try and return a datetime object
+ d = dateparser(x, dayfirst=dayfirst, yearfirst=yearfirst)
+ return d
+ mydateparser = with_default_value(mydateparser, datetime.datetime(1,1,1,0,0,0))
@tacaswell

tacaswell Mar 19, 2016

Owner

This line has multiple pep8 violations (too long and lacking spaces after ,).

Owner

tacaswell commented Mar 19, 2016

Can you add a test to test_mlab.py which exercises this as well?

Owner

tacaswell commented Mar 19, 2016

This is also a candidate for backporting to 1.5.x

Contributor

ultra-andy commented Mar 20, 2016

Thanks @tacaswell!

Ok, I missed pep8, I understand what the violations are and can fix them.

I presume that I need to close the commit, then after modifying the code resubmit it - no drama.

I've had a look at adding a test to test_mlab.py, but have no idea how to do it - for instance, the test object should be a .csv file, so it isn't as simple as calling csv2rec with a few arguments, and then checking that the result is as expected. Should I create a csv file with the test values in it within test_mlab.py (ie, manually create .csv file on the spot), then open this just-created file and check the values read out? I also don't understand the overall structure or mechanism of the test_mlab.py file or of unit tests for matplotlib, so is there any resource out there explaining how to add tests? I don't see any reference to existing csv2rec tests in there, so this looks like a new initiative, at least for csv2rec.

I'll look at backporting once we've worked out all the above (!).

Any guidance would be helpful - thanks.

Contributor

ultra-andy commented Mar 20, 2016

Ok, found the csv2rec tests in line 305 onwards of test_mlab.py. Now trying to understand them - they follow the model of creating temporary files, then testing that the results returned are as expected.

Member

WeatherGod commented Mar 20, 2016

You don't need to close the PR, just keep pushing new commits to your
branch. The PR will automatically update itself.

On Sun, Mar 20, 2016 at 3:20 AM, ultra-andy notifications@github.com
wrote:

Ok, found the csv2rec tests in line 305 onwards of test_mlab.py. Now
trying to understand them - they follow the model of creating temporary
files, then testing that the results returned are as expected.


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#6185 (comment)

ultra-andy added some commits Mar 20, 2016

@ultra-andy ultra-andy Creation of fix to #6184 - csv2rec bug
yearfirst & dayfirst date interpretation bug for datetime strings -
csv2rec does not take yearfirst or dayfirst into account when
interpreting datetime strings with a time component, so for ambiguous
cases, the month and day get transposed. Test added also.
bc38288
@ultra-andy ultra-andy forgot the library...!
adding in datetime library import
3d1d043
@ultra-andy ultra-andy more PEP-8 fixes...
removal of spaces around "="
9ad1dc2
@ultra-andy ultra-andy PEP-8 fix
addition of extra blank line
5f7b119
Contributor

ultra-andy commented Mar 20, 2016

Great, the auto tests now pass!

Assuming that's all OK and gets approved, now how do I backport this fix to 1.5.x? I'm using the Windows GitHub interface, I've created a branch off 1.5.x, but have no idea where the 1.5.x files are or should be ??

Guidance appreciated...

Owner

tacaswell commented Mar 20, 2016

After this is merged we cherry-pick the merge commit back to the 1.5.x branch. You don't need to do anything extra.

Contributor

ultra-andy commented Mar 20, 2016

Excellent!

A few questions about my workflow - is it OK to rely on the auto checks after submission to ensure adherence to PEP-8, or is this poor form?

Also, any comments on code will be happily received - I don't have a lot of Python experience, so my code may have obvious improvements.

Owner

tacaswell commented Mar 20, 2016

It is best if you run the test locally before pushing (if nothing else, it helps to make your iteration loop much faster).

I strongly suggest setting up a linter in what ever editor you use. PEP8 compliance can be annoying, but having a consistent style does help reading and understanding parts of the code you are not familiar with. Given the size of the mpl code base and the number of unique contributors we have we need to have some sort of consistent style and by picking PEP8 we fit in with everyone else and some one else writes the validation tools 😜 .

Owner

tacaswell commented Mar 20, 2016

The code in this PR is fine. It could have also been done via a lambda or partial, but a local closure is just as clear

@tacaswell tacaswell added a commit that referenced this pull request Mar 20, 2016

@tacaswell tacaswell Merge pull request #6185 from ultra-andy/branch-csv2rec-datetime-dayf…
…irst-bug

FIX: csv2rec bug

Closes #6184
b64bd25

@tacaswell tacaswell merged commit b64bd25 into matplotlib:master Mar 20, 2016

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

tacaswell removed the needs_review label Mar 20, 2016

@tacaswell tacaswell added a commit that referenced this pull request Mar 20, 2016

@tacaswell tacaswell Merge pull request #6185 from ultra-andy/branch-csv2rec-datetime-dayf…
…irst-bug

FIX: csv2rec bug

Closes #6184
a919ac7
Owner

tacaswell commented Mar 20, 2016

backported to v1.5.x as a919ac7

Contributor

ultra-andy commented Mar 21, 2016

Got it - I tried PyLint (integrated into PyScripter) and found it far stricter than the PEP-8 standard, then found autopep8, which runs at the command line, and that works very nicely. I'll try that next time prior to commit (if I ever find another bug annoying enough for me to want to fix it!)

Just so I understand the background process a bit better, what makes you decide when something should go into 1.5.2, and when something should be held over for 2.0.1? And when do these get released?

ultra-andy deleted the ultra-andy:branch-csv2rec-datetime-dayfirst-bug branch Mar 21, 2016

Owner

tacaswell commented Mar 21, 2016

low-risk bug fixes get backported to the maintenance branch (current 1.5.x).

The next feature release is planned to be 2.0 (with major style changes) and should be released as soon as we finish it.

1.5.2 will only be released if we need to, but we are accumulating bug-fixes on the 1.5.x branch just-in-case.

@tacaswell tacaswell added a commit to tacaswell/matplotlib that referenced this pull request May 22, 2016

@tacaswell tacaswell Merge pull request #6185 from ultra-andy/branch-csv2rec-datetime-dayf…
…irst-bug

FIX: csv2rec bug

Closes #6184
63a6fee
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment