The datasize of Shuffle read and write is much larger

[zuoliwei]

hello, when I run the wordcount tesk in Hibench ,datasize is 5g, I found that the datasize of shuffle write and read using firestorm(Memory_HDFS mode) is much larger than using the external shuffle service with the same spark parameters

could you please analyze why it happened

[colinmjj]

@zuoliwei currently, Firestorm haven't function of map side combine, maybe the problem is caused by this.
with Firestorm, you can get benefit in following situation:

1. Support spark on K8S better
2. Performance improvement when random IO seriously with ESS
   In your test case, I think performance degradation with Firestorm is in expectation

[zuoliwei]

OK ，I got it thank you