[SPARK-20177]Document about compression way has some little detail ch… #17498
Conversation
docs/configuration.md
Outdated
| </td> | ||
| </tr> | ||
| <tr> | ||
| <td><code>spark.io.compression.codec</code></td> | ||
| <td>lz4</td> | ||
| <td> | ||
| The codec used to compress internal data such as RDD partitions, broadcast variables and | ||
| shuffle outputs. By default, Spark provides three codecs: <code>lz4</code>, <code>lzf</code>, | ||
| The codec used to compress internal data such as RDD partitions,event log, broadcast variables |
There was a problem hiding this comment.
I asked a few people, they all don't know, the event log compression methods is spark.io.com pression. Codec.And a few other places it is necessary to modify, more friendly and clear.
There was a problem hiding this comment.
I'm not sure what you're replying to, but I'm asking you to add the missing space in "partitions,event"
|
IMHO I thought this is still a not so necessary fix. I would doubt if user really get confused without your fix? You can always correct me since I stand on the of developers :). |
|
@srowen i add a space |
|
@jerryshao |
|
@srowen @jerryshao |
|
Test build #3632 has finished for PR 17498 at commit
|
|
Merged to master |
…anges.
What changes were proposed in this pull request?
Document compression way little detail changes.
1.spark.eventLog.compress add 'Compression will use spark.io.compression.codec.'
2.spark.broadcast.compress add 'Compression will use spark.io.compression.codec.'
3,spark.rdd.compress add 'Compression will use spark.io.compression.codec.'
4.spark.io.compression.codec add 'event log describe'.
eg
Through the documents, I don't know what is compression mode about 'event log'.
How was this patch tested?
manual tests
Please review http://spark.apache.org/contributing.html before opening a pull request.