Add IndexedSet to help manage workers/jobs.

[stephenh]

This is admittedly somewhat cute, but I like the idea of more strictly/DRYly applying the "remove the job from all xxxToJob indexes" logic, instead of having to manually remember to do the "xxxToJob -= job" calls.

[mateiz]

This is definitely interesting. Let me think about it a bit more (haven't had a lot of time in the past few days), but it might be worth going for if we can use this throughout.

[stephenh]

Yesterday I had a job run into a race condition in StandaloneSchedulerBackend where freeCores was being decremented in StatusUpdate, but the executor had already been removed, so it failed with NoSuchElementException:

13/02/17 07:03:23 ERROR cluster.StandaloneSchedulerBackend$DriverActor: key not found: 13
java.util.NoSuchElementException: key not found: 13
  at scala.collection.MapLike$class.default(MapLike.scala:225)
  at scala.collection.mutable.HashMap.default(HashMap.scala:45)
  at scala.collection.MapLike$class.apply(MapLike.scala:135)
  at scala.collection.mutable.HashMap.apply(HashMap.scala:45)
  at spark.scheduler.cluster.StandaloneSchedulerBackend$DriverActor$$anonfun$receive$1.apply(StandaloneSchedulerBackend.scala:60)

I thought this would be a good excuse for more IndexedSet cuteness, so fixed the bug by having just one "executors" map and ensuring the executor still exists before updating it.

Also merged in master and ran the tests.

[stephenh]

Again remerged in master with your job->app/split->partition changes (nice!).

[stephenh]

I believe this change is right--we only have 1 new executor, so I believe calling makeOffers(e) is fine, vs. the previous behavior which would re-makeOffers() for the new + all existing executors.

[AmplabJenkins]

Can one of the admins verify this patch?

[AmplabJenkins]

I'm the Jenkins test bot for the UC, Berkeley AMPLab. I've noticed your pull request and will test it once an admin authorizes me to. Thanks for your submission!

[AmplabJenkins]

I'm the Jenkins test bot for the UC, Berkeley AMPLab. I've noticed your pull request and will test it once an admin authorizes me to. Thanks for your submission!

[AmplabJenkins]

Thank you for your pull request. An admin will review this request soon.