[SPARK-5061][Alex Baretta] SQLContext: overload createParquetFile

[alexbaretta]

Overload of createParquetFile taking a StructType instead of a TypeTag

[AmplabJenkins]

Can one of the admins verify this patch?

[ash211]

Jenkins this is ok to test

[ash211]

I kind of think createEmptyParquetFile would be a better name for this method, since most Parquet files have data I'd think

[alexbaretta]

Andrew,

OK, but keep in mind that my patch overloads an existing method. If you
think createParquetFile should be renamed to createEmptyParquetFile you
should probably file a separate JIRA.

Also, arguably "creating a file" implies that it is empty.

Alex
On Jan 2, 2015 5:11 PM, "Andrew Ash" notifications@github.com wrote:

In sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala
#3882 (diff):

• * val schema = StructType(List(StructField("name", StringType),StructField("age", IntegerType)))
• * createParquetFile(schema, "path/to/file.parquet").registerTempTable("people")
• * sql("INSERT INTO people SELECT 'michael', 29")
• * }}}
• * @param schema StructType describing the records to be stored in the Parquet file.
• * @param path The path where the directory containing parquet metadata should be created.
• * Data inserted into this table will also be stored at this location.
• * @param allowExisting When false, an exception will be thrown if this directory already exists.
• * @param conf A Hadoop configuration object that can be used to specify options to the parquet
• * output format.
• * @group userf
• */
• @experimental
• def createParquetFile(

I kind of think createEmptyParquetFile would be a better name for this
method, since most Parquet files have data I'd think

—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/3882/files#r22428199.

[SparkQA]

Test build #25000 has started for PR 3882 at commit f6e40b5.

• This patch merges cleanly.

[SparkQA]

Test build #25000 has finished for PR 3882 at commit f6e40b5.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25000/
Test FAILed.

[alexbaretta]

In retrospect amending my commit might not have been the right thing to do... Any feedback on how to properly amend a PR would be appreciated.