-
Notifications
You must be signed in to change notification settings - Fork 226
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Better handle rows that break across splits, and other small related fixes #400
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not super happy about the hack here, but I could not find any other way around it. This is a potential correctness issue, so we need to do something. At least, I added more tests to exercise handling of rows that split across splits, and they still pass.
reader = new InputStreamReader(in, charset) | ||
|
||
if (codec == null) { | ||
// Hack: in the uncompressed case (see more below), we must know how much the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea ... I don't like this hack too ... but seems no better way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good if the styles and tests pass
Codecov Report
@@ Coverage Diff @@
## master #400 +/- ##
==========================================
- Coverage 87.78% 87.73% -0.06%
==========================================
Files 14 14
Lines 745 758 +13
Branches 64 65 +1
==========================================
+ Hits 654 665 +11
- Misses 91 93 +2
Continue to review full report at Codecov.
|
Thanks @HyukjinKwon - we may want to cut an 0.6.0 release for this, plus the |
This attempt to address #398
See also #399
The change is I believe explained in comments below.