[SPARK-8619][Streaming]Don't recover keytab and principal configuration within Streaming checkpoint

[SaintBacchus]

Client.scala will change these configurations, so this would cause the problem that the Streaming recover logic can't find the local keytab file(since configuration was changed)

      sparkConf.set("spark.yarn.keytab", keytabFileName)
      sparkConf.set("spark.yarn.principal", args.principal)

Problem described at Jira

[SparkQA]

Test build #35749 has finished for PR 7008 at commit 0d8f800.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class StreamingLinearAlgorithm(object):
  • class StreamingLogisticRegressionWithSGD(StreamingLinearAlgorithm):
  • case class CountFunction(expr: Expression, base: AggregateExpression) extends AggregateFunction
  • case class CountDistinctFunction(
  • case class ApproxCountDistinctPartitionFunction(
  • case class ApproxCountDistinctMergeFunction(
  • case class Sum(child: Expression) extends PartialAggregate with trees.UnaryNode[Expression]
  • case class CombineSum(child: Expression) extends AggregateExpression
  • case class SumDistinct(child: Expression)
  • case class CombineSetsAndSum(inputSet: Expression, base: Expression) extends AggregateExpression
  • case class CombineSetsAndSumFunction(
  • case class First(child: Expression) extends PartialAggregate with trees.UnaryNode[Expression]
  • case class Last(child: Expression) extends PartialAggregate with trees.UnaryNode[Expression]
  • case class PrecisionInfo(precision: Int, scale: Int)

[harishreedharan]

You don't really need to reload the principal, but that is not a big deal. This LGTM.

We probably should document that these parameters will not be reloaded from checkpoint, so the conf should still have the correct params (not sure if these params were documented in the first place).

@tgravescs - this looks good to go.

[SaintBacchus]

@harishreedharan You are right, the principal is all the same.
I considerred them as couple configurations, so I added it into the list.:smile:

[harishreedharan]

@tgravescs, @tdas - This might be worth getting into a 1.4.1 if another RC is rolled. We should get this into 1.4.2 either way.

[tgravescs]

changes look good to me. I'll let @tdas or other streaming maintainer look at commit this though as I'm not as familiar with this code.

[andrewor14]

can you call this propertiesToReload? It's confusing to have reloadConfs here and newReloadConf down there when they're not describing the same thing.

[andrewor14]

also, please add a short comment here to explain why this is necessary.

[SparkQA]

Test build #36065 has finished for PR 7008 at commit 9b8e92c.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #36080 has finished for PR 7008 at commit d50dbdf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

OK, LGTM I'll merge this into master and 1.4.

[SparkQA]

Test build #36170 has finished for PR 7008 at commit d50dbdf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tdas]

LGTM! Merging this to master and 1.4. Thanks!