Skip to content

Commit

Permalink
[SPARK-4194] [core] Make SparkContext initialization exception-safe.
Browse files Browse the repository at this point in the history
SparkContext has a very long constructor, where multiple things are
initialized, multiple threads are spawned, and multiple opportunities
for exceptions to be thrown exist. If one of these happens at an
innoportune time, lots of garbage tends to stick around.

This patch re-organizes SparkContext so that its internal state is
initialized in a big "try" block. The fields keeping state are now
completely private to SparkContext, and are "vars", because Scala
doesn't allow you to initialize a val later. The existing API interface
is kept by turning vals into defs (which works because Scala guarantees
the same binary interface for those).

On top of that, a few things in other areas were changed to avoid more
things leaking:

- Executor was changed to explicitly wait for the heartbeat thread to
  stop. LocalBackend was changed to wait for the "StopExecutor"
  message to be received, since otherwise there could be a race
  between that message arriving and the actor system being shut down.
- ConnectionManager could possibly hang during shutdown, because an
  interrupt at the wrong moment could cause the selector thread to
  still call select and then wait forever. So also wake up the
  selector so that this situation is avoided.
  • Loading branch information
Marcelo Vanzin committed Apr 2, 2015
1 parent 071f16e commit 8caa8b3
Show file tree
Hide file tree
Showing 6 changed files with 320 additions and 234 deletions.
Loading

0 comments on commit 8caa8b3

Please sign in to comment.