Move compressionCodec to a string parameter #174
Conversation
|
Instead of changing |
Current coverage is
|
|
@msperlich would you please add a method named |
| * org.apache.hadoop.io.compress.CompressionCodec then the output will be | ||
| * compressed. | ||
| */ | ||
| def saveAsCsvFile(path: String, parameters: Map[String, String]): Unit = { |
There was a problem hiding this comment.
I prefer not to add this method. saveAsCsfFile is a helper method for convenience. This functionality should be implemented in `CsvParser``
|
@falaki can we please deprecate CsvContext? We should go through the unified reader interface, which is consistent across all languages. CsvContext should only be used for backward compatibility. |
|
@falaki I've tried to rework this PR to match what you've described. Let me know how it looks now. I'm new to both Scala and Spark, so thanks for your patience! |
|
any followup on this one? |
| @@ -26,6 +26,16 @@ package object csv { | |||
| val defaultCsvFormat = | |||
| CSVFormat.DEFAULT.withRecordSeparator(System.getProperty("line.separator", "\n")) | |||
|
|
|||
| private[csv] def compresionCodecClass(className: String): Class[_ <: CompressionCodec] = { | |||
| className match { | |||
| case null => null | |||
This change allows Python (and presumably Java) users to use compressed output by passing the compressionCodec as a string option, instead of an argument to a function only visible in the implicit class. This is an API breaking change, so I'm definitely open to other approaches.
Add overloaded versions of saveAsCsvFile to preserve API compatibilty. Fixed checkstyle violations.
Got rid of overloaded methods, added codec as a parameter to CsvParser. Updated README with some examples.
Remove extra spaces, fix indentation.
|
Thanks. Merging this now for release 1.3.0 |
This change allows Python (and presumably Java) users to use compressed
output by passing the compressionCodec as a string option, instead of an
argument to a function only visible in the implicit class.
This is an API breaking change, so I'm definitely open to other
approaches.