Chapter 3: java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.

[kumgaurav]

Created a scala program and trying to run the streaming code. it is filing with below error

19/11/09 07:25:24 ERROR StreamExecution: Query customer_purchases_2 [id = 6863c8d1-fd1c-49eb-a454-92825f0d2782, runId = 8adbd1e6-04e1-4b79-9fad-05950f359e77] terminated with error
java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
This stopped SparkContext was created at:

org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:901)
spark_guide.chapter3.StructuredStreaming$.delayedEndpoint$spark_guide$chapter3$StructuredStreaming$1(StructuredStreaming.scala:12)
spark_guide.chapter3.StructuredStreaming$delayedInit$body.apply(StructuredStreaming.scala:9)
scala.Function0$class.apply$mcV$sp(Function0.scala:34)
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
scala.App$$anonfun$main$1.apply(App.scala:76)
scala.App$$anonfun$main$1.apply(App.scala:76)
scala.collection.immutable.List.foreach(List.scala:392)
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
scala.App$class.main(App.scala:76)
spark_guide.chapter3.StructuredStreaming$.main(StructuredStreaming.scala:9)
spark_guide.chapter3.StructuredStreaming.main(StructuredStreaming.scala)

[WilliamCVan]

To fix that bug add this line after the writeStream.start() command, it will keep the sparkContext open for you to see the results. I was writing to console, was not using in memory table fyi.

spark.streams.awaitAnyTermination()