Parallelism in Spark and tuning
What do you need to know about ๐ฆ๐ฝ๐ฎ๐ฟ๐ธ ๐ฃ๐ฎ๐ฟ๐ฎ๐น๐น๐ฒ๐น๐ถ๐๐บ?
While optimizing Spark Applications you will usually tweak two elements - performance and resource utilization.
Understanding parallelism in Spark and tuning it according to the situation will help you in both.
๐ฆ๐ผ๐บ๐ฒ ๐๐ฎ๐ฐ๐๐:
โก๏ธ Spark Executor can have multiple CPU Cores assigned to it. โก๏ธ Number of CPU Cores per Spark executor is defined by ๐๐ฝ๐ฎ๐ฟ๐ธ.๐ฒ๐ ๐ฒ๐ฐ๐๐๐ผ๐ฟ.๐ฐ๐ผ๐ฟ๐ฒ๐ configuration. โก๏ธ Single CPU Core can read one file or partition of a splittable file at a single point in time. โก๏ธ Once read a file is transformed into one or multiple partitions in memory.
๐ข๐ฝ๐๐ถ๐บ๐ถ๐๐ถ๐ป๐ด ๐ฅ๐ฒ๐ฎ๐ฑ ๐ฃ๐ฎ๐ฟ๐ฎ๐น๐น๐ฒ๐น๐ถ๐๐บ:
โ๏ธ If number of cores is equal to the number of files, files are not splittable and some of them are larger in size - larger files become a bottleneck, Cores responsible for reading smaller files will idle for some time. โ๏ธ If there are more Cores than the number of files - Cores that do not have files assigned to them will Idle. If we do not perform repartition after reading the files - the cores will remain Idle during processing stages.
โ Rule of thumb: set number of Cores to be two times less than files being read. Adjust according to your situation.
๐ข๐ฝ๐๐ถ๐บ๐ถ๐๐ถ๐ป๐ด ๐ฃ๐ฟ๐ผ๐ฐ๐ฒ๐๐๐ถ๐ป๐ด ๐ฃ๐ฎ๐ฟ๐ฎ๐น๐น๐ฒ๐น๐ถ๐๐บ:
โก๏ธ Use ๐๐ฝ๐ฎ๐ฟ๐ธ.๐ฑ๐ฒ๐ณ๐ฎ๐๐น๐.๐ฝ๐ฎ๐ฟ๐ฎ๐น๐น๐ฒ๐น๐ถ๐๐บ and ๐๐ฝ๐ฎ๐ฟ๐ธ.๐๐พ๐น.๐๐ต๐๐ณ๐ณ๐น๐ฒ.๐ฝ๐ฎ๐ฟ๐๐ถ๐๐ถ๐ผ๐ป๐ configurations to set the number of partitions created after performing wide transformations. โก๏ธ After reading the files there will be as many partitions as there were files or partitions in splittable files.
โ๏ธ After data is loaded as partitions into memory - Spark jobs will suffer from the same set of parallelism inefficiencies like when reading the data.
โ Rule of thumb: set ๐๐ฝ๐ฎ๐ฟ๐ธ.๐ฑ๐ฒ๐ณ๐ฎ๐๐น๐.๐ฝ๐ฎ๐ฟ๐ฎ๐น๐น๐ฒ๐น๐ถ๐๐บ equal to ๐๐ฝ๐ฎ๐ฟ๐ธ.๐ฒ๐ ๐ฒ๐ฐ๐๐๐ผ๐ฟ.๐ฐ๐ผ๐ฟ๐ฒ๐ times the number of executors times a small number from 2 to 8, tune to specific Spark job.
๐๐ฑ๐ฑ๐ถ๐๐ถ๐ผ๐ป๐ฎ๐น ๐ก๐ผ๐๐ฒ๐:
๐ You can use ๐๐ฝ๐ฎ๐ฟ๐ธ.๐๐พ๐น.๐ณ๐ถ๐น๐ฒ๐.๐บ๐ฎ๐ ๐ฃ๐ฎ๐ฟ๐๐ถ๐๐ถ๐ผ๐ป๐๐๐๐ฒ๐ configuration to set maximum size of the partition when reading files. Files that are larger will be split into multiple partitions accordingly. ๐ It has been shown that write throughput starts to bottleneck once there are more than 5 CPU Cores assigned per Executor so keep ๐๐ฝ๐ฎ๐ฟ๐ธ.๐ฒ๐ ๐ฒ๐ฐ๐๐๐ผ๐ฟ.๐ฐ๐ผ๐ฟ๐ฒ๐ at or below 5.