Woken / Spark cannot start under OSX

[ludovicc]

While running ./tests.sh, the experiment test fails and we can see in the logs

17/08/17 14:35:38 INFO Utils: Successfully started service 'sparkDriver' on port 34333.
17/08/17 14:35:38 INFO SparkEnv: Registering MapOutputTracker
17/08/17 14:35:38 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: System memory 466092032 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
	at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:212)
	at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:194)
	at org.apache.spark.SparkEnv$.create(SparkEnv.scala:308)
	at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:165)
	at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:259)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:423)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2256)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:831)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:823)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:823)
	at eu.hbp.mip.woken.core.validation.Scores$class.$init$(Scores.scala:24)
	at eu.hbp.mip.woken.core.validation.RegressionScores.<init>(Scores.scala:185)
	at eu.hbp.mip.woken.core.validation.Scores$.apply(Scores.scala:49)
	at eu.hbp.mip.woken.core.CrossValidationActor$$anonfun$15.applyOrElse(coordinator.scala:635)
	at eu.hbp.mip.woken.core.CrossValidationActor$$anonfun$15.applyOrElse(coordinator.scala:564)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
	at akka.actor.FSM$class.processEvent(FSM.scala:604)
	at eu.hbp.mip.woken.core.CrossValidationActor.akka$actor$LoggingFSM$$super$processEvent(coordinator.scala:510)
	at akka.actor.LoggingFSM$class.processEvent(FSM.scala:734)
	at eu.hbp.mip.woken.core.CrossValidationActor.processEvent(coordinator.scala:510)
	at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:598)
	at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:592)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
	at eu.hbp.mip.woken.core.CrossValidationActor.aroundReceive(coordinator.scala:510)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)

[ERROR] [08/17/2017 14:35:38.188] [woken-akka.actor.default-dispatcher-4] [akka://woken/user/$n/$a/$b] System memory 466092032 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
java.lang.IllegalArgumentException: System memory 466092032 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
	at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:212)
	at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:194)
	at org.apache.spark.SparkEnv$.create(SparkEnv.scala:308)
	at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:165)
	at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:259)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:423)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2256)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:831)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:823)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:823)
	at eu.hbp.mip.woken.core.validation.Scores$class.$init$(Scores.scala:24)
	at eu.hbp.mip.woken.core.validation.RegressionScores.<init>(Scores.scala:185)
	at eu.hbp.mip.woken.core.validation.Scores$.apply(Scores.scala:49)
	at eu.hbp.mip.woken.core.CrossValidationActor$$anonfun$15.applyOrElse(coordinator.scala:635)
	at eu.hbp.mip.woken.core.CrossValidationActor$$anonfun$15.applyOrElse(coordinator.scala:564)

[ludovicc]

Seen also:

 [WARN] [08/18/2017 14:54:57.854] [woken-akka.remote.default-remote-dispatcher-7] [akka.tcp://woken@woken:8088/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fwoken%40woken-validation%3A8082-0] Association with remote system [akka.tcp://woken@woken-validation:8082] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://woken@woken-validation:8082]] Caused by: [woken-validation]