Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
[SPARK-9439] [yarn] External shuffle service robust to NM restarts using leveldb #7943
In general, Yarn apps should be robust to NodeManager restarts. However, if you run spark with the external shuffle service on, after a NM restart all shuffles fail, b/c the shuffle service has lost some state with info on each executor. (Note the shuffle data is perfectly fine on disk across a NM restart, the problem is we've lost the small bit of state that lets us find those files.)
The solution proposed here is that the external shuffle service can write out its state to leveldb (backed by a local file) every time an executor is added. When running with yarn, that file is in the NM's local dir. Whenever the service is started, it looks for that file, and if it exists, it reads the file and re-registers all executors there.
Nothing is changed in non-yarn modes with this patch. The service is not given a place to save the state to, so it operates the same as before. This should make it easy to update other cluster managers as well, by just supplying the right file & the equivalent of yarn's
This does not seem to work properly.
After the first nodemanager restart:
Workaround: switch back to spark-1.5.2-yarn.shuffle.jar
There is a workaround that you can apply waiting to have this pull request merged. You will need to check that path set on the yarn-site.xml configuration yarn.nodemanager.local-dirs does not start with spaces, return charrier... or is not a URI (file:///data/yarn/ won't work, this will work /data/yarn/)