Small doc change on how to use dynamic partitioning in Spark SQL

[markgrover]

No description provided.

[SparkQA]

Test build #47492 has finished for PR 10248 at commit 8684706.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * case class LambdaVariable(value: String, isNull: String, dataType: DataType) extends LeafExpression\n

[srowen]

This isn't quite a feature like the other elements in the list. Can it be a shorter elaboration on the previous point? or is this better documented elsewhere rather than in just the table of contents, so to speak?

[markgrover]

Sure thing, I have updated it to be a part of the previous bullet. Open to making a separate heading too but I think it may be an overkill since there will only be one sentence under that heading. Thanks for reviewing, Sean!

[SparkQA]

Test build #47558 has finished for PR 10248 at commit d299637.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Does this (also) belong in the scaladoc somewhere? Maybe to slip this in better, it's implicit that you use the same Hive syntax so that can be skipped, and just parenthetically mention the setting you must make to use it? I know this is trivial, so don't mind merging as is either, just trying to make it fit a bit better.

[andrewor14]

@yhuai @liancheng

[markgrover]

There isn't really an API where this gets called from, it's simple how you'd write that SQL query so I don't know of a good place to add it in scaladoc.

Thanks again for reviewing, Sean.

[yhuai]

What do you mean by mentioning SET hive.exec.dynamic.partition.mode=nonstrict?

[markgrover]

In Hive or in Spark SQL, as I understand, the default partitioning mode is
strict, due to which at least one of the partitions being inserted to has
to be statically specified. When doing dynamic partitioning, this mode has
to be changed before. In spark SQL, that means you have to issue a query
like sqlContext.sql("SET hive.exec.dynamic.partition.mode=nonstrict")
before running the actual SQL statement.

Does that answer your question?

On Wed, Dec 16, 2015 at 4:16 PM, Yin Huai notifications@github.com wrote:

What do you mean by mentioning SET
hive.exec.dynamic.partition.mode=nonstrict?

—
Reply to this email directly or view it on GitHub
#10248 (comment).

[yhuai]

I feel it can be confusing to users because users can insert into a Hive partitioned table and they can also create partitioned tables backed by data source API. For data source API backed tables, there is no need to use this conf. For inserting into a Hive table, users can set it to either strict or nonstrict mode. I am not sure we need to add this change.

[markgrover]

Ok, thanks Yin. I was just trying to be helpful. I feel like we have spent
enough collective cycles on this and if it's not a no brainer, it's
probably not worth it:-)
Thanks Yin and Sean for your feedback, I will close the PR.

On Wed, Dec 16, 2015 at 8:27 PM, Yin Huai notifications@github.com wrote:

I feel it can be confusing to users because users can insert into a Hive
partitioned table and they can also create partitioned tables backed by
data source API. For data source API backed tables, there is no need to use
this conf. For inserting into a Hive table, users can set it to either
strict or nonstrict mode. I am not sure we need to add this change.

—
Reply to this email directly or view it on GitHub
#10248 (comment).