You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Jan 9, 2020. It is now read-only.
Currently, Spark jobs use dirs inside the driver and executor pods for storing temporary files. For instance, the work dirs for the Spark driver and executors use dirs inside the pods. And internal shuffle service per executor also uses in-pod dirs.
These in-pod dirs are within the docker storage backend, which can be slow due to its copy-on-write overhead. Many of the storage backends implement block level CoW. Each small write will incur copy of the entire block. The overhead can become very high if the files are updated by many small writes. It is recommended to avoid using docker storage backend for such use cases. From the first link above:
Ideally, very little data is written to a container’s writable layer, and you use Docker volumes to write data.
We should use EmptyDir for temporary storage to avoid this overhead.
The text was updated successfully, but these errors were encountered:
This suggestion seems very reasonable as a potential performance improvement. Good suggestion @kimoonkim !
Ideally we could look at some benchmarks before/after of some Spark jobs to compare the difference. And they would need to be shuffle heavy to emphasize the differences on this write path.
Currently, Spark jobs use dirs inside the driver and executor pods for storing temporary files. For instance, the work dirs for the Spark driver and executors use dirs inside the pods. And internal shuffle service per executor also uses in-pod dirs.
These in-pod dirs are within the docker storage backend, which can be slow due to its copy-on-write overhead. Many of the storage backends implement block level CoW. Each small write will incur copy of the entire block. The overhead can become very high if the files are updated by many small writes. It is recommended to avoid using docker storage backend for such use cases. From the first link above:
We should use
EmptyDir
for temporary storage to avoid this overhead.The text was updated successfully, but these errors were encountered: