[SPARK-3569][SQL] Add metadata field to StructField #2701

mengxr · 2014-10-08T00:24:34Z

Add metadata: Metadata to StructField to store extra information of columns. Metadata is a simple wrapper over Map[String, Any] with value types restricted to Boolean, Long, Double, String, Metadata, and arrays of those types. SerDe is via JSON.

Metadata is preserved through simple operations like SELECT.

@marmbrus @liancheng

SparkQA · 2014-10-08T00:30:02Z

QA tests have started for PR 2701 at commit d8af0ed.

This patch does not merge cleanly!

SparkQA · 2014-10-08T01:09:55Z

QA tests have started for PR 2701 at commit c41a664.

This patch merges cleanly.

SparkQA · 2014-10-08T01:22:52Z

QA tests have finished for PR 2701 at commit c41a664.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class AttributeReference(
- case class StructField(

AmplabJenkins · 2014-10-08T01:22:55Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21435/Test FAILed.

SparkQA · 2014-10-08T01:39:42Z

QA tests have finished for PR 2701 at commit d8af0ed.

This patch fails unit tests.
This patch does not merge cleanly!

AmplabJenkins · 2014-10-08T01:39:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21429/Test FAILed.

liancheng · 2014-10-08T01:48:26Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala

-    nameToField.get(name).getOrElse(
-      throw new IllegalArgumentException(s"Field ${name} does not exist."))
+    nameToField.getOrElse(name,
+      throw new IllegalArgumentException(s"Field $name does not exist."))


Nit: no need to wrap

liancheng · 2014-10-08T02:02:42Z

I think using immutable Map for metadata is enough. We can add an API like .transformMetadata(f: Map[String, Any] => Map[String, Any]) to alter metadata as needed.

liancheng · 2014-10-08T02:44:10Z

PySpark also need to be updated.

SparkQA · 2014-10-08T02:49:34Z

QA tests have started for PR 2701 at commit 7e5a322.

This patch merges cleanly.

mengxr · 2014-10-08T02:53:26Z

@liancheng The Python API is hard to update at this time because the schema SerDe is via str(..):

https://github.com/apache/spark/blob/master/python/pyspark/sql.py#L1131

SparkQA · 2014-10-08T03:35:32Z

QA tests have finished for PR 2701 at commit 7e5a322.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class AttributeReference(
- case class StructField(

AmplabJenkins · 2014-10-08T03:35:39Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21440/Test PASSed.

liancheng · 2014-10-08T03:53:47Z

#2563 has already replaced str(...) with .json(). You can add Python metadata ser/de by modifying .jsonValue().

SparkQA · 2014-10-09T07:49:39Z

QA tests have started for PR 2701 at commit 93518fb.

This patch merges cleanly.

SparkQA · 2014-10-09T09:00:24Z

QA tests have finished for PR 2701 at commit 93518fb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class AttributeReference(
- case class StructField(

AmplabJenkins · 2014-10-09T09:00:27Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21529/Test FAILed.

SparkQA · 2014-10-21T05:34:49Z

QA tests have started for PR 2701 at commit 611d3c2.

This patch merges cleanly.

SparkQA · 2014-10-21T07:14:14Z

QA tests have finished for PR 2701 at commit 611d3c2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class AttributeReference(
- case class StructField(
- class MetadataBuilder

AmplabJenkins · 2014-10-21T07:14:18Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21967/
Test PASSed.

davies · 2014-10-23T23:06:32Z

python/pyspark/sql.py

@@ -305,12 +305,15 @@ class StructField(DataType):

    """

-    def __init__(self, name, dataType, nullable):
+    def __init__(self, name, dataType, nullable, metadata={}):


Use {} as default value will have side effects, such as:

>>> a = StructField('a', StringType(), True) >>> b = StructField('b', StringType(), True) >>> a.metadata['name'] = 'a' >>> b.metadata['name'] 'a'

So if the metadata could be modified somewhere, here you should use None as default value.

def xxx(xxx, metadata=None): .... self.metadata = metadata or {}

SparkQA · 2014-10-24T00:00:09Z

QA tests have started for PR 2701 at commit 589f314.

This patch merges cleanly.

SparkQA · 2014-10-24T01:33:29Z

QA tests have finished for PR 2701 at commit 589f314.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class AttributeReference(
- case class StructField(
- class MetadataBuilder

AmplabJenkins · 2014-10-24T01:33:33Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22098/
Test PASSed.

SparkQA · 2014-10-25T08:10:01Z

Test build #443 has started for PR 2701 at commit 589f314.

This patch merges cleanly.

SparkQA · 2014-10-25T09:32:59Z

Test build #443 has finished for PR 2701 at commit 589f314.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class AttributeReference(
- case class StructField(
- class MetadataBuilder

SparkQA · 2014-10-26T22:06:13Z

Test build #458 has started for PR 2701 at commit 589f314.

This patch merges cleanly.

SparkQA · 2014-10-26T22:33:03Z

Test build #465 has started for PR 2701 at commit 589f314.

This patch merges cleanly.

SparkQA · 2014-10-26T23:37:22Z

Test build #458 has finished for PR 2701 at commit 589f314.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class AttributeReference(
- case class StructField(
- class MetadataBuilder

marmbrus · 2014-10-27T00:03:29Z

@mengxr, thanks for working on this! Overall LGTM. One minor thing: I think we should expose Metadata as a type variable in the org.apache.spark.sql package and as a subclass in org.apache.spark.sql.api.java. Catalyst is considered a Private API so we explicitly expose the parts of it in sql that we expect users to use.

SparkQA · 2014-10-27T00:12:05Z

Test build #465 has finished for PR 2701 at commit 589f314.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class AttributeReference(
- case class StructField(
- class MetadataBuilder

… packages.

marmbrus · 2014-10-29T02:04:46Z

Here's a PR to fix the package visibility. If that looks good to you I think this is ready to merge: mengxr#1

Expose Metadata and MetadataBuilder through the public scala and java packages.

SparkQA · 2014-10-30T19:00:23Z

QA tests have started for PR 2701 at commit c35203f.

This patch does not merge cleanly.

SparkQA · 2014-10-30T20:17:52Z

QA tests have finished for PR 2701 at commit c35203f.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
- case class AttributeReference(
- case class StructField(
- class MetadataBuilder
- class Metadata extends org.apache.spark.sql.catalyst.util.Metadata
- public class MetadataBuilder extends org.apache.spark.sql.catalyst.util.MetadataBuilder

AmplabJenkins · 2014-10-30T20:17:55Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22543/
Test FAILed.

SparkQA · 2014-11-01T01:20:11Z

Test build #22671 has started for PR 2701 at commit dedda56.

This patch merges cleanly.

SparkQA · 2014-11-01T02:55:24Z

Test build #22671 has finished for PR 2701 at commit dedda56.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class AttributeReference(
- case class StructField(
- class MetadataBuilder
- class Metadata extends org.apache.spark.sql.catalyst.util.Metadata
- public class MetadataBuilder extends org.apache.spark.sql.catalyst.util.MetadataBuilder

AmplabJenkins · 2014-11-01T02:55:28Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22671/
Test PASSed.

marmbrus · 2014-11-01T21:37:25Z

Thanks! Merged to master.

mengxr added 5 commits September 17, 2014 14:41

add metadata field to StructField and Attribute

c194d5e

add test

367d237

remove Map.empty

d65072e

add test on join

67fdebb

move tests to SQLQuerySuite

d8af0ed

merge master

c41a664

liancheng reviewed Oct 8, 2014
View reviewed changes

do not output metadata in StructField.toString

7e5a322

mengxr added 4 commits October 8, 2014 22:59

merge master

61b8e0f

make tests work in scala

618e349

java conversions

905bb89

support metadata in python

93518fb

mengxr changed the title ~~[WIP][SPARK-3569][SQL] Add metadata field to StructField~~ [SPARK-3569][SQL] Add metadata field to StructField Oct 9, 2014

liancheng mentioned this pull request Oct 9, 2014

[SPARK-3809][SQL] Fixes test suites in hive-thriftserver #2675

Closed

davies reviewed Oct 23, 2014
View reviewed changes

mengxr added 2 commits October 23, 2014 16:55

change default value of metadata to None in python

1e2abcf

Merge remote-tracking branch 'apache/master' into structfield-metadata

589f314

Expose Metadata and MetadataBuilder through the public scala and java…

886b85c

… packages.

Merge pull request #1 from marmbrus/pr/2701

c35203f

Expose Metadata and MetadataBuilder through the public scala and java packages.

mengxr added 2 commits October 31, 2014 18:09

Merge remote-tracking branch 'apache/master' into structfield-metadata

5ef930a

merge remote

dedda56

asfgit closed this in 1d4f355 Nov 1, 2014

[SPARK-3569][SQL] Add metadata field to StructField #2701

[SPARK-3569][SQL] Add metadata field to StructField #2701

Conversation

mengxr commented Oct 8, 2014

SparkQA commented Oct 8, 2014

SparkQA commented Oct 8, 2014

SparkQA commented Oct 8, 2014

AmplabJenkins commented Oct 8, 2014

SparkQA commented Oct 8, 2014

AmplabJenkins commented Oct 8, 2014

liancheng Oct 8, 2014

Choose a reason for hiding this comment

liancheng commented Oct 8, 2014

liancheng commented Oct 8, 2014

SparkQA commented Oct 8, 2014

mengxr commented Oct 8, 2014

SparkQA commented Oct 8, 2014

AmplabJenkins commented Oct 8, 2014

liancheng commented Oct 8, 2014

SparkQA commented Oct 9, 2014

SparkQA commented Oct 9, 2014

AmplabJenkins commented Oct 9, 2014

SparkQA commented Oct 21, 2014

SparkQA commented Oct 21, 2014

AmplabJenkins commented Oct 21, 2014

davies Oct 23, 2014

Choose a reason for hiding this comment

SparkQA commented Oct 24, 2014

SparkQA commented Oct 24, 2014

AmplabJenkins commented Oct 24, 2014

SparkQA commented Oct 25, 2014

SparkQA commented Oct 25, 2014

SparkQA commented Oct 26, 2014

SparkQA commented Oct 26, 2014

SparkQA commented Oct 26, 2014

marmbrus commented Oct 27, 2014

SparkQA commented Oct 27, 2014

marmbrus commented Oct 29, 2014

SparkQA commented Oct 30, 2014

SparkQA commented Oct 30, 2014

AmplabJenkins commented Oct 30, 2014

SparkQA commented Nov 1, 2014

SparkQA commented Nov 1, 2014

AmplabJenkins commented Nov 1, 2014

marmbrus commented Nov 1, 2014