[SPARK-20434][YARN][CORE] Move Hadoop delegation token code from yarn to core #17723

mgummelt · 2017-04-21T21:05:50Z

What changes were proposed in this pull request?

Move Hadoop delegation token code from spark-yarn to spark-core, so that other schedulers (such as Mesos), may use it. In order to avoid exposing Hadoop interfaces in spark-core, the new Hadoop delegation token classes are kept private. In order to provider backward compatiblity, and to allow YARN users to continue to load their own delegation token providers via Java service loading, the old YARN interfaces, as well as the client code that uses them, have been retained.

Summary:

Move registered yarn.security.ServiceCredentialProvider classes from spark-yarn to spark-core. Moved them into a new, private hierarchy under HadoopDelegationTokenProvider. Client code in HadoopDelegationTokenManager now loads credentials from a whitelist of three providers (HadoopFSDelegationTokenProvider, HiveDelegationTokenProvider, HBaseDelegationTokenProvider), instead of service loading, which means that users are not able to implement their own delegation token providers, as they are in the spark-yarn module.
The yarn.security.ServiceCredentialProvider interface has been kept for backwards compatibility, and to continue to allow YARN users to implement their own delegation token provider implementations. Client code in YARN now fetches tokens via the new YARNHadoopDelegationTokenManager class, which fetches tokens from the core providers through HadoopDelegationTokenManager, as well as service loads them from yarn.security.ServiceCredentialProvider.

Old Hierarchy:

yarn.security.ServiceCredentialProvider (service loaded)
  HadoopFSCredentialProvider
  HiveCredentialProvider
  HBaseCredentialProvider
yarn.security.ConfigurableCredentialManager

New Hierarchy:

HadoopDelegationTokenManager
HadoopDelegationTokenProvider (not service loaded)
  HadoopFSDelegationTokenProvider
  HiveDelegationTokenProvider
  HBaseDelegationTokenProvider

yarn.security.ServiceCredentialProvider (service loaded)
yarn.security.YARNHadoopDelegationTokenManager

How was this patch tested?

unit tests

…e generic SparkHadoopUtil class

…r for backwards compatibility

…eCredentialProvider from core

SparkQA · 2017-04-21T21:08:37Z

Test build #76044 has finished for PR 17723 at commit ad4e33b.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-04-21T21:18:41Z

Test build #76045 has finished for PR 17723 at commit e15f1ab.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-04-21T21:29:05Z

Test build #76046 has finished for PR 17723 at commit a546aab.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

mgummelt · 2017-04-21T22:39:30Z

core/pom.xml

+      -->
+    <dependency>
+      <groupId>${hive.group}</groupId>
+      <artifactId>hive-exec</artifactId>


I still don't know how to place these in the test scope, which is where they belong. See my comment here: https://github.com/apache/spark/pull/17665/files#r112337820

So I tried a few things, and the one that got me further was just having this:

<dependency> <groupId>${hive.group}</groupId> <artifactId>hive-metastore</artifactId> <scope>test</scope> </dependency>

And nix the others. Adding the others in test scope caused some weird error in sbt, even with all dependencies (we have the dependencies you had problems with cached locally).

My comment was going to be to add that, then rewrite the code to use the metastore API instead of the Hive class from hive-exec... but then I noticed that test is not doing much, because there are no metastore servers to talk to. It's even there, hardcoded in the test:

test("obtain tokens For HiveMetastore") { ... credentials.getAllTokens.size() should be (0)

All it seems to be doing is making sure the reflection-based code is not completely broken. That is something already, though.

So I have two suggestions, in order of preference:

add the dependencies in "provided" scope, and change the code to use actual types and not reflection. Because the classes may not exist at runtime, that means having to handle NoClassDefFoundError in creative ways.

keep the reflection code, and remove this test. Or maybe move it to a separate module as others have suggested.

I kinda like the first because it's always good to avoid reflection, and this is a particularly ugly use of it.

Thanks for looking into it.

Do you know why reflection was used in the first place? Why not just add the Hive dependencies to compile scope? I'm thinking that's what we should do now, and drop reflection.

So I'm agreeing with your first bullet point, but proposing that we add the hive deps to compile rather than provided.

Why not just add the Hive dependencies to compile scope?

Because technically Hive is an optional dependency for Spark, and moving it to compile scope would break that.

(Whether that should change or not is a separate discussion, but probably better not to have it as part of this change.)

Alright I added hive-exec to provided scope, and removed the reflection.

SparkQA · 2017-04-22T00:33:43Z

Test build #76047 has finished for PR 17723 at commit d6d21d1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mgummelt · 2017-04-22T00:38:24Z

cc @vanzin @jerryshao @skonto

BTW @vanzin, I decided to parameterize HadoopFSCredentialProvider with a new HadoopAccessManager object, for which YARN provides a custom YARNHadoopAccessManager. I did this instead of conditioning on SparkHadoopUtil.get.isYarnMode, since I prefer functional parameterization over global values.

SparkQA · 2017-06-09T00:30:47Z

Test build #77820 has finished for PR 17723 at commit 7e2f90d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-09T00:49:13Z

Test build #77822 has finished for PR 17723 at commit 563b80a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-06-09T06:59:42Z

Thanks! Will review it tomorrow.

vanzin

Few style nits remaining, otherwise LGTM.

vanzin · 2017-06-09T22:37:53Z

core/pom.xml

@@ -357,6 +357,34 @@
      <groupId>org.apache.commons</groupId>
      <artifactId>commons-crypto</artifactId>
    </dependency>
+
+    <!--
+     Testing Hive reflection needs hive on the test classpath only, however, while adding hive-exec to the test


Minor, but this comment is a little stale now after all the changes. There's no reflection anymore, so the "provided" scope is just so we can still package Spark without Hive.

vanzin · 2017-06-09T22:39:31Z

core/src/main/scala/org/apache/spark/deploy/security/HadoopFSCredentialProvider.scala

+  }
+
+  private def fetchDelegationTokens(
+    renewer: String,


nit: indent extra level

vanzin · 2017-06-09T22:39:44Z

core/src/main/scala/org/apache/spark/deploy/security/HadoopFSCredentialProvider.scala

+  }
+
+  private def getTokenRenewalInterval(
+    hadoopConf: Configuration,


nit: indent extra level

vanzin · 2017-06-09T22:40:48Z

core/src/main/scala/org/apache/spark/deploy/security/HiveCredentialProvider.scala

+          creds.addToken(new Text("hive.server2.delegation.token"), hive2Token)
+        }
+      } catch {
+        case NonFatal(e) =>


Move this catch block to the end of the method?

good catch. fixed.

vanzin · 2017-06-09T22:42:43Z

...src/main/scala/org/apache/spark/deploy/yarn/security/YARNConfigurableCredentialManager.scala

+  // public for testing
+  val credentialProviders = getCredentialProviders
+
+  def obtainCredentials(


nit: fits in one line.

vanzin · 2017-06-09T22:43:07Z

...src/main/scala/org/apache/spark/deploy/yarn/security/YARNConfigurableCredentialManager.scala

+      hadoopConf: Configuration,
+      creds: Credentials): Long = {
+
+    val superInterval = configurableCredentialManager.obtainCredentials(


nit: fits in one line.

vanzin · 2017-06-09T22:43:43Z

...src/main/scala/org/apache/spark/deploy/yarn/security/YARNConfigurableCredentialManager.scala

+  }
+
+  private def loadCredentialProviders: List[ServiceCredentialProvider] = {
+    ServiceLoader.load(


nit: the load(...) call fits in one line.

fixed. would be nice if scalastyle caught all of these spacing issues.

gatorsmile

In the PR description, we need an update. Although this PR does not have a design doc, we still need to clearly describe all the major changes and the relation between these new classes and the existing/new traits.

For example, the relation between ServiceCredentialProvider and HadoopDelegationTokenProvider. Another example, [[HadoopFSCredentialProvider]], [[HiveCredentialProvider]] and [[HBaseCredentialProvider]] needs to be explained too.

If possible, draw a class hierarchy for them. It will help the future code readers to understand the impact of this PR, after we merge it.

gatorsmile · 2017-06-09T19:54:48Z

core/src/main/scala/org/apache/spark/deploy/security/ConfigurableCredentialManager.scala

+      }
+    }
+
+    val isEnabledDeprecated = deprecatedProviderEnabledConfigs.forall { pattern =>


Why? It would be a pure, 0-ary function, which is better represented as a val.

This will be used only when key is not defined in SparkConf.

gatorsmile · 2017-06-09T19:58:15Z

core/src/main/scala/org/apache/spark/deploy/security/HBaseCredentialProvider.scala

 import org.apache.spark.internal.Logging
 import org.apache.spark.util.Utils

-private[security] class HBaseCredentialProvider extends ServiceCredentialProvider with Logging {
+private[security] class HBaseCredentialProvider
+  extends HadoopDelegationTokenProvider with Logging {


Nit: Shorten it to one line

It's too long now with the rename.

gatorsmile · 2017-06-09T20:17:34Z

core/src/main/scala/org/apache/spark/deploy/security/HadoopDelegationTokenProvider.scala

+private[spark] trait HadoopDelegationTokenProvider {
+
+  /**
+   * Name of the service to provide credentials. This name should be unique, Spark internally will


Nit: unique, -> unique.

gatorsmile · 2017-06-09T20:21:18Z

core/src/main/scala/org/apache/spark/deploy/security/HadoopFSCredentialProvider.scala

+
+private[deploy] class HadoopFSCredentialProvider(fileSystems: Set[FileSystem])
+    extends HadoopDelegationTokenProvider with Logging {
+  // Token renewal interval, this value will be set in the first call,


How about?

The token renewal interval will be set in the first call.

gatorsmile · 2017-06-09T21:37:15Z

core/src/main/scala/org/apache/spark/deploy/security/HiveCredentialProvider.scala

+
+  override def serviceName: String = "hive"
+
+  private val classNotFoundErrorStr = "You are attempting to use the HiveCredentialProvider," +


Nit: ," -> , "

gatorsmile · 2017-06-09T21:54:40Z

resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala

@@ -368,7 +371,9 @@ private[spark] class Client(
    val fs = destDir.getFileSystem(hadoopConf)

    // Merge credentials obtained from registered providers
-    val nearestTimeOfNextRenewal = credentialManager.obtainCredentials(hadoopConf, credentials)
+    val nearestTimeOfNextRenewal = credentialManager.obtainCredentials(


Revert this back?

gatorsmile · 2017-06-10T19:21:45Z

...src/main/scala/org/apache/spark/deploy/yarn/security/YARNConfigurableCredentialManager.scala

+  // public for testing
+  val credentialProviders = getCredentialProviders
+
+  def obtainCredentials(


YARNConfigurableCredentialManager is not extending ConfigurableCredentialManager . Please add the function description for this function too.

gatorsmile · 2017-06-10T19:25:04Z

...src/main/scala/org/apache/spark/deploy/yarn/security/YARNConfigurableCredentialManager.scala

+    hadoopConf: Configuration,
+    fileSystems: Set[FileSystem]) extends Logging {
+
+  private val configurableCredentialManager =


Add the comment above this line to explain why YARNConfigurableCredentialManager does not extend ConfigurableCredentialManager

I'm not sure what you mean. Is composition vs. inheritance something that needs to be justified?

gatorsmile · 2017-06-10T19:39:34Z

core/src/main/scala/org/apache/spark/deploy/security/ConfigurableCredentialManager.scala

+  /**
+   * Get credential provider for the specified service.
+   */
+  def getServiceCredentialProvider(service: String): Option[HadoopDelegationTokenProvider] = {


In this ConfigurableCredentialManager , we are using the terminology ServiceCredentialProvider. However, ServiceCredentialProvider is a Yarn-specific trait. It is confusing when reading the codes.

If possible, we need to change the names and terms used in the class ConfigurableCredentialManager

You're right. I made a few changes to naming:

*CredentialProvider -> *DelegationTokenProvider
ConfigurableCredentialManager -> HadoopDelegationTokenManager
YARNConfigurableCredentialManager -> YARNHadoopDelegationTokenManager

and updated a bunch of comments.

cc @vanzin. you might be interested in this since these renames are non-trivial

gatorsmile · 2017-06-10T19:48:29Z

core/src/main/scala/org/apache/spark/deploy/security/ConfigurableCredentialManager.scala

+   * Writes delegation tokens to creds.  Delegation tokens are fetched from all registered
+   * providers.
+   *
+   * @return Time after which the fetched delegation tokens should be renewed.


This needs to be more accurate to explain the first service provider that needs to renew.

I think this is the most accurate and succinct explanation of the contract. Since we aren't returning the renewal time of all tokens, it is true that after the returned timeout, all tokens must be renewed. I could say "Time after which one of the returned tokens must be renewed", but this is a circuitous instruction to the user, since they actually must renew all.

mgummelt · 2017-06-12T18:20:23Z

@gatorsmile I've expanded the description, and included a before/after class hierarchy.

SparkQA · 2017-06-12T20:20:56Z

Test build #77941 has finished for PR 17723 at commit c684d88.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

mgummelt · 2017-06-12T20:58:53Z

retest this please

SparkQA · 2017-06-12T23:12:13Z

Test build #77946 has finished for PR 17723 at commit c684d88.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

mgummelt · 2017-06-12T23:19:37Z

retest this please

SparkQA · 2017-06-13T01:36:52Z

Test build #77953 has finished for PR 17723 at commit c684d88.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

mgummelt · 2017-06-13T17:07:09Z

@gatorsmile All comments addressed. Ready for re-review. Unit tests are failing on master, so I'm waiting for those to be resolved to get a passing build.

mgummelt · 2017-06-14T18:04:34Z

retest this please

gatorsmile · 2017-06-14T20:45:34Z

docs/running-on-yarn.md

@@ -504,10 +504,10 @@ spark.yarn.access.hadoopFileSystems hdfs://ireland.example.org:8020/,webhdfs://f
 ```

 Spark supports integrating with other security-aware services through Java Services mechanism (see
-`java.util.ServiceLoader`). To do that, implementations of `org.apache.spark.deploy.yarn.security.ServiceCredentialProvider`
+`java.util.ServiceLoader`). To do that, implementations of `org.apache.spark.deploy.security.ServiceCredentialProvider`


revert it back?

gatorsmile · 2017-06-14T20:51:03Z

LGTM except one comment.

cc @vanzin @mridulm @cloud-fan @rxin

SparkQA · 2017-06-14T20:59:40Z

Test build #78062 has finished for PR 17723 at commit c684d88.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-15T03:23:13Z

Test build #78072 has finished for PR 17723 at commit c4149dd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2017-06-15T18:45:40Z

LGTM, merging to master.

mgummelt · 2017-06-15T20:23:56Z

w00t

Thanks everyone.

… to core ## What changes were proposed in this pull request? Move Hadoop delegation token code from `spark-yarn` to `spark-core`, so that other schedulers (such as Mesos), may use it. In order to avoid exposing Hadoop interfaces in spark-core, the new Hadoop delegation token classes are kept private. In order to provider backward compatiblity, and to allow YARN users to continue to load their own delegation token providers via Java service loading, the old YARN interfaces, as well as the client code that uses them, have been retained. Summary: - Move registered `yarn.security.ServiceCredentialProvider` classes from `spark-yarn` to `spark-core`. Moved them into a new, private hierarchy under `HadoopDelegationTokenProvider`. Client code in `HadoopDelegationTokenManager` now loads credentials from a whitelist of three providers (`HadoopFSDelegationTokenProvider`, `HiveDelegationTokenProvider`, `HBaseDelegationTokenProvider`), instead of service loading, which means that users are not able to implement their own delegation token providers, as they are in the `spark-yarn` module. - The `yarn.security.ServiceCredentialProvider` interface has been kept for backwards compatibility, and to continue to allow YARN users to implement their own delegation token provider implementations. Client code in YARN now fetches tokens via the new `YARNHadoopDelegationTokenManager` class, which fetches tokens from the core providers through `HadoopDelegationTokenManager`, as well as service loads them from `yarn.security.ServiceCredentialProvider`. Old Hierarchy: ``` yarn.security.ServiceCredentialProvider (service loaded) HadoopFSCredentialProvider HiveCredentialProvider HBaseCredentialProvider yarn.security.ConfigurableCredentialManager ``` New Hierarchy: ``` HadoopDelegationTokenManager HadoopDelegationTokenProvider (not service loaded) HadoopFSDelegationTokenProvider HiveDelegationTokenProvider HBaseDelegationTokenProvider yarn.security.ServiceCredentialProvider (service loaded) yarn.security.YARNHadoopDelegationTokenManager ``` ## How was this patch tested? unit tests Author: Michael Gummelt <mgummelt@mesosphere.io> Author: Dr. Stefan Schimanski <sttts@mesosphere.io> Closes apache#17723 from mgummelt/SPARK-20434-refactor-kerberos.

… to core (Kerberos refactor) Move Hadoop delegation token code from `spark-yarn` to `spark-core`, so that other schedulers (such as Mesos), may use it. In order to avoid exposing Hadoop interfaces in spark-core, the new Hadoop delegation token classes are kept private. In order to provider backward compatiblity, and to allow YARN users to continue to load their own delegation token providers via Java service loading, the old YARN interfaces, as well as the client code that uses them, have been retained. Summary: - Move registered `yarn.security.ServiceCredentialProvider` classes from `spark-yarn` to `spark-core`. Moved them into a new, private hierarchy under `HadoopDelegationTokenProvider`. Client code in `HadoopDelegationTokenManager` now loads credentials from a whitelist of three providers (`HadoopFSDelegationTokenProvider`, `HiveDelegationTokenProvider`, `HBaseDelegationTokenProvider`), instead of service loading, which means that users are not able to implement their own delegation token providers, as they are in the `spark-yarn` module. - The `yarn.security.ServiceCredentialProvider` interface has been kept for backwards compatibility, and to continue to allow YARN users to implement their own delegation token provider implementations. Client code in YARN now fetches tokens via the new `YARNHadoopDelegationTokenManager` class, which fetches tokens from the core providers through `HadoopDelegationTokenManager`, as well as service loads them from `yarn.security.ServiceCredentialProvider`. Old Hierarchy: ``` yarn.security.ServiceCredentialProvider (service loaded) HadoopFSCredentialProvider HiveCredentialProvider HBaseCredentialProvider yarn.security.ConfigurableCredentialManager ``` New Hierarchy: ``` HadoopDelegationTokenManager HadoopDelegationTokenProvider (not service loaded) HadoopFSDelegationTokenProvider HiveDelegationTokenProvider HBaseDelegationTokenProvider yarn.security.ServiceCredentialProvider (service loaded) yarn.security.YARNHadoopDelegationTokenManager ``` unit tests Author: Michael Gummelt <mgummelt@mesosphere.io> Author: Dr. Stefan Schimanski <sttts@mesosphere.io> Closes apache#17723 from mgummelt/SPARK-20434-refactor-kerberos.

Dr. Stefan Schimanski and others added 20 commits April 17, 2017 17:50

[Mesosphere SPARK-126] Move YarnSparkHadoopUtil token helpers into th…

ce63a9b

…e generic SparkHadoopUtil class

[Mesosphere SPARK-126] Add Mesos Kerberos support

75d849a

Par down kerberos support

35002f2

cleanup

13981c8

style

af4a3e4

Add MesosSecurityManager

5cc66dc

info logs

a47c9c0

style

c8ec049

Re-add org.apache.spark.deploy.yarn.security.ServiceCredentialProvide…

954eeff

…r for backwards compatibility

move YARNHadoopFSCredentialProviderSuite

2d76928

Move hive test deps to the core module

d8a968d

remove test scope

b8093c8

remove test scope

25d5088

Removed MesosSecurityManager, added RPC call, removed META-INF Servic…

4c387eb

…eCredentialProvider from core

add InterfaceStability annotation to ServiceCredentialProvider

e32afee

Add HadoopAccessManager

be69f5a

Remove mesos code

55616da

re-add mistakenly removed files

240df31

test ConfigurableCredentialManager.obtainUserTokens

810c6b2

add tests

ad4e33b

rat-excludes

e15f1ab

fix RAT

a546aab

style

d6d21d1

mgummelt commented Apr 21, 2017

View reviewed changes

Remove unneeded import

092aac7

vanzin reviewed Jun 9, 2017

View reviewed changes

gatorsmile reviewed Jun 10, 2017

View reviewed changes

rename

c684d88

mgummelt changed the title ~~[SPARK-20434][YARN][CORE] Move kerberos delegation token code from yarn to core~~ [SPARK-20434][YARN][CORE] Move Hadoop delegation token code from yarn to core Jun 12, 2017

gatorsmile reviewed Jun 14, 2017

View reviewed changes

fix docs

c4149dd

asfgit closed this in a18d637 Jun 15, 2017

mgummelt deleted the SPARK-20434-refactor-kerberos branch June 15, 2017 20:48

vanzin mentioned this pull request Jul 13, 2017

[SPARK-21377][YARN] Make jars specify with --jars/--packages load-able in AM's credential renwer #18616

Closed

ifilonenko mentioned this pull request Jul 25, 2017

Secure HDFS Support requires most recent PRs apache-spark-on-k8s/spark#390

Open

ArtRand restored the SPARK-20434-refactor-kerberos branch August 21, 2017 20:13


		override def serviceName: String = "hive"

		private val classNotFoundErrorStr = "You are attempting to use the HiveCredentialProvider," +

[SPARK-20434][YARN][CORE] Move Hadoop delegation token code from yarn to core #17723

[SPARK-20434][YARN][CORE] Move Hadoop delegation token code from yarn to core #17723

Conversation

mgummelt commented Apr 21, 2017 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented Apr 21, 2017

SparkQA commented Apr 21, 2017

SparkQA commented Apr 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mgummelt Jun 2, 2017 • edited Loading

Choose a reason for hiding this comment

SparkQA commented Apr 22, 2017

mgummelt commented Apr 22, 2017

SparkQA commented Jun 9, 2017

SparkQA commented Jun 9, 2017

gatorsmile commented Jun 9, 2017

vanzin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gatorsmile left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mgummelt Jun 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mgummelt commented Jun 12, 2017

SparkQA commented Jun 12, 2017

mgummelt commented Jun 12, 2017

SparkQA commented Jun 12, 2017

mgummelt commented Jun 12, 2017

SparkQA commented Jun 13, 2017

mgummelt commented Jun 13, 2017

mgummelt commented Jun 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gatorsmile commented Jun 14, 2017

SparkQA commented Jun 14, 2017

SparkQA commented Jun 15, 2017

vanzin commented Jun 15, 2017

mgummelt commented Jun 15, 2017

mgummelt commented Apr 21, 2017 •

edited

Loading

mgummelt Jun 2, 2017 •

edited

Loading

mgummelt Jun 12, 2017 •

edited

Loading