[SPARK-9486][SQL] Add data source aliasing for external packages

[JDrit]

Users currently have to provide the full class name for external data sources, like:

sqlContext.read.format("com.databricks.spark.avro").load(path)

This allows external data source packages to register themselves using a Service Loader so that they can add custom alias like:

sqlContext.read.format("avro").load(path)

This makes it so that using external data source packages uses the same format as the internal data sources like parquet, json, etc.

[SparkQA]

Test build #1306 has finished for PR 7802 at commit 208a2a8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • trait DataSourceProvider
  • trait RelationProvider extends DataSourceProvider
  • trait SchemaRelationProvider extends DataSourceProvider
  • trait HadoopFsRelationProvider extends DataSourceProvider
  • trait CreatableRelationProvider extends DataSourceProvider

[vanzin]

Perhaps just add META-INF/services/ instead? For future-proofness.

[vanzin]

Looks ok as far as I can tell. I generally find it weird to use traits for public APIs (too easy to break compatibility), but then all the API here is scala, so maybe it's not a big deal. I also wonder if there's a test that can be written to ensure that we're not mistakenly registering two sources with the same name.

And finally, ServiceLoader may have some funny semantics when used with userClassPathFirst, given the current implementation of ChildFirstURLClassLoader. Might be worth it to make a note to look at that behavior in more detail.

[JDrit]

I looked into the userClassPathFirst and it did not seem to interfere with anything. I turned it on for both the driver and executor and all the tests still passed.

[JDrit]

I moved the classloader out to a lazy val as well.

[JDrit]

There are also test to ensure correct functionality for when two or more data sources are registered under the same name.

[rxin]

@JDrit what's still WIP about this patch?

[rxin]

Actually I think the current API breaks binary compatibility for data sources, so we can't merge it as is.

In Java (or Scala binary compatibility), RelationProvider now has an extra interface that has no default implementation. We need to find a workaround to provide this information.

[SparkQA]

Test build #1345 has finished for PR 7802 at commit 74db85e.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • sys.error(s"Failed to load class for data source: $provider")
  • trait DataSourceProvider
  • trait RelationProvider extends DataSourceProvider
  • trait SchemaRelationProvider extends DataSourceProvider
  • trait HadoopFsRelationProvider extends DataSourceProvider
  • trait CreatableRelationProvider extends DataSourceProvider

[JDrit]

@rxin I changed the interface that provided the alias to be a mixin used in the different data sources, so that should fix the binary compatibility problem. Data sources now mixin this trait if they want to provide an alias for themselves. Let me know if this satisfies your concerns.

[SparkQA]

Test build #1372 has finished for PR 7802 at commit 87b7f1c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class In(value: Expression, list: Seq[Expression]) extends Predicate
  • case class InSet(child: Expression, hset: Set[Any]) extends UnaryExpression with Predicate
  • .getOrElse(sys.error(s"Failed to load class for data source: $provider"))
  • trait DataSourceRegister

[SparkQA]

Test build #1378 has finished for PR 7802 at commit 87b7f1c.

• This patch fails Spark unit tests.
• This patch does not merge cleanly.
• This patch adds the following public classes (experimental):
  • .getOrElse(sys.error(s"Failed to load class for data source: $provider"))
  • trait DataSourceRegister

[rxin]

@JDrit still failing orc.

[JDrit]

It was an issue with the class loader not being reloaded on every call of lookupDataSource, this fix should fix that.

[vanzin]

Can this be META-INF/services/*? I can see someone creating a package with actual source files called services.

[SparkQA]

Test build #1395 has finished for PR 7802 at commit 72b349a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • sys.error(s"Failed to load class for data source: $provider")
  • trait DataSourceRegister

[SparkQA]

Test build #1398 has finished for PR 7802 at commit e5e93b2.

• This patch fails Spark unit tests.
• This patch does not merge cleanly.
• This patch adds the following public classes (experimental):
  • class BlockMatrix(DistributedMatrix):
  • case class In(value: Expression, list: Seq[Expression]) extends Predicate
  • case class InSet(child: Expression, hset: Set[Any]) extends UnaryExpression with Predicate
  • sys.error(s"Failed to load class for data source: $provider")
  • trait DataSourceRegister

[chenghao-intel]

tryLoad(loader, s"$provider.DefaultSource") => tryLoad(loader, s"$provider")?

[chenghao-intel]

Oh, sorry, it's both supported.

[SparkQA]

Test build #1402 timed out for PR 7802 at commit e5e93b2 after a configured wait of 175m.

[SparkQA]

Test build #1411 has finished for PR 7802 at commit e5e93b2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • sys.error(s"Failed to load class for data source: $provider")
  • trait DataSourceRegister

[rxin]

I'm going to merge this - I will submit a pr later to change the API slightly.

[tedyu]

Should orc be added as well ?
I see change to OrcRelation.scala below.

[JDrit]

Orc is added in the other resource file since hive is a sperate package.