New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
added support for saving with a QuoteMode #254
Conversation
Current coverage is
|
val carsCopy = sqlContext.csvFile(copyFilePath + "/") | ||
|
||
assert(carsCopy.count == cars.count) | ||
assert(carsCopy.collect.map(_.toString).toSet == cars.collect.map(_.toString).toSet) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm.. Does this really check if the output is quoted or not?
Not sure why it's not rerunning the tests, but I pushed extended tests. I'm not that familiar with Scala, so please feel free to write that in a more elegant way. |
@HyukjinKwon Are you interested in merging this in? Do you have more feedback? |
@tobithiel I do not have the permission. cc @falaki |
@tobithiel This is a cool feature to have in spark-csv. Would you make sure it works with the other parse mode as well? |
@falaki I'm not sure I get what you mean? This only deals with writing files and since parsing csv files with quotes is already tested & works, parsing the written files again works out of the box. |
@tobithiel Thanks. This looks good. |
I'm dealing with some messy csv files and being able to just quote all fields is very useful, so that other applications don't misunderstand the file because of some sketchy characters.
This also enables disabling quoting all together, which would resolve #119, and only quoting non-numeric fields.