Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-9148][SPARK-10252][SQL] Update SQL Programming Guide #8441

Closed
wants to merge 4 commits into from
Closed

[SPARK-9148][SPARK-10252][SQL] Update SQL Programming Guide #8441

wants to merge 4 commits into from

Conversation

marmbrus
Copy link
Contributor

No description provided.

@SparkQA
Copy link

SparkQA commented Aug 26, 2015

Test build #41581 has finished for PR 8441 at commit f11c169.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class FPGrowthModel[Item: ClassTag] @Since("1.3.0") (
    • class FreqItemset[Item] @Since("1.3.0") (
    • class FreqSequence[Item] @Since("1.5.0") (
    • class PrefixSpanModel[Item] @Since("1.5.0") (

rxin
rxin reviewed Aug 26, 2015
View reviewed changes
docs/sql-programming-guide.md
new data.

<table class="table">
<tr><th>Scala/Java</th><th>Python</th><th>Meaning</th></tr>
<tr><th>Scala/Java</th><th>Any Language</th><th>Meaning</th></tr>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python/R rather than Any Language?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works in scala too though? I wasn't really sure what to do here. We could also just delete the other column?

@cloud-fan
Copy link
Contributor

In JDBC part we give an example of how to include extra jars by SPARK_CLASSPATH=postgresql-9.3-1102-jdbc41.jar bin/spark-shell. However, SPARK_CLASSPATH is deprecated, should we update that?

@tgravescs
Copy link
Contributor

Perhaps this is a separate jira but the build doc needs to be updated for building with hive also.

tgravescs
tgravescs reviewed Aug 26, 2015
View reviewed changes
docs/sql-programming-guide.md
<li>A classpath in the standard format for both Hive and Hadoop.</li>
Use Hive jars of specified version downloaded from Maven repositories. This configuration
is not generally recommended for production deployments.
<li>A classpath in the standard format for the JVM. This classpath must include all of Hive
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be nice if we could say something about the jars either need to be installed on the cluster or on yarn shipped with your application.

@SparkQA
Copy link

SparkQA commented Aug 27, 2015

Test build #41663 has finished for PR 8441 at commit 4ca2b96.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor

rxin commented Aug 27, 2015

LGTM

tgravescs
tgravescs reviewed Aug 27, 2015
View reviewed changes
docs/sql-programming-guide.md
is not generally recommended for production deployments.
<li>A classpath in the standard format for the JVM. This classpath must include all of Hive
and its dependencies, including the correct version of Hadoop. These jars only need to be
present on the driver, but if you are running in yarn client mode then you must ensure
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These jars aren't needed by the executors at all? If that is the case the only time they need to be shipped is in yarn cluster mode.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, they are only used by the driver to get metadata. Thanks for the clarification on cluster vs client.

@tgravescs
Copy link
Contributor

thanks LGTM

@SparkQA
Copy link

SparkQA commented Aug 27, 2015

Test build #41704 has finished for PR 8441 at commit f3fdf62.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

asfgit pushed a commit that referenced this pull request Aug 27, 2015
@marmbrus
[SPARK-9148] [SPARK-10252] [SQL] Update SQL Programming Guide
965b3bb 
Author: Michael Armbrust <michael@databricks.com>

Closes #8441 from marmbrus/documentation.

(cherry picked from commit dc86a22)
Signed-off-by: Michael Armbrust <michael@databricks.com>
@asfgit asfgit closed this in dc86a22 Aug 27, 2015
@marmbrus
Copy link
Contributor Author

Thanks guys! Merged to master and 1.5

@marmbrus marmbrus deleted the documentation branch March 8, 2016 00:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
5 participants
@marmbrus @SparkQA @cloud-fan @tgravescs @rxin