Skip to content

[SPARK-16870][docs]Summary:add "spark.sql.broadcastTimeout" into docs/sql-programming-gu…#14477

Closed
biglobster wants to merge 2 commits intoapache:masterfrom
biglobster:keliang
Closed

[SPARK-16870][docs]Summary:add "spark.sql.broadcastTimeout" into docs/sql-programming-gu…#14477
biglobster wants to merge 2 commits intoapache:masterfrom
biglobster:keliang

Conversation

@biglobster
Copy link
Contributor

What changes were proposed in this pull request?

default value for spark.sql.broadcastTimeout is 300s. and this property do not show in any docs of spark. so add "spark.sql.broadcastTimeout" into docs/sql-programming-guide.md to help people to how to fix this timeout error when it happenned

How was this patch tested?

not need

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

…ide.md

JIRA_ID:SPARK-16870
Description:default value for spark.sql.broadcastTimeout is 300s. and this property do not show in any docs of spark. so add "spark.sql.broadcastTimeout" into docs/sql-programming-guide.md to help people to how to fix this timeout error when it happenned
Test:done

…ide.md

JIRA_ID:SPARK-16870
Description:default value for spark.sql.broadcastTimeout is 300s. and this property do not show in any docs of spark. so add "spark.sql.broadcastTimeout" into docs/sql-programming-guide.md to help people to how to fix this timeout error when it happenned
Test:done
@srowen
Copy link
Member

srowen commented Aug 3, 2016

Many properties are not documented on purpose, because they're semi-private. What's the argument that this was intended to be public and set by callers?

@biglobster
Copy link
Contributor Author

biglobster commented Aug 3, 2016

hi, @srowen
here my workload and what I found
I run a large number jobs with spark-sql at the same time. and meet the error that print timeout (some job contains the broadcast-join operator) :
java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:107)
at org.apache.spark.sql.execution.joins.BroadcastHashOuterJoin.doExecute(BroadcastHashOuterJoin.scala:113)
so if I reset value bigger then 300s, error do not show again

@srowen
Copy link
Member

srowen commented Aug 4, 2016

Any objection to documenting it @liancheng ?

</p>
</td>
</tr>
<tr>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PS this does not look like the right place for this option. This section covers Parquet. There's a later "other options" section covering broadcast join. Move it there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done. thx :) for you suggestion

JIRA_ID:SPARK-16870
Description:Move "spark.sql.broadcastTimeout" to broadcast join secsion
Test:done
@srowen
Copy link
Member

srowen commented Aug 7, 2016

Jenkins test this please

@SparkQA
Copy link

SparkQA commented Aug 7, 2016

Test build #63322 has finished for PR 14477 at commit 52911bf.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

asfgit pushed a commit that referenced this pull request Aug 7, 2016
…s/sql-programming-gu…

## What changes were proposed in this pull request?
default value for spark.sql.broadcastTimeout is 300s. and this property do not show in any docs of spark. so add "spark.sql.broadcastTimeout" into docs/sql-programming-guide.md to help people to how to fix this timeout error when it happenned

## How was this patch tested?

not need

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

…ide.md

JIRA_ID:SPARK-16870
Description:default value for spark.sql.broadcastTimeout is 300s. and this property do not show in any docs of spark. so add "spark.sql.broadcastTimeout" into docs/sql-programming-guide.md to help people to how to fix this timeout error when it happenned
Test:done

Author: keliang <keliang@cmss.chinamobile.com>

Closes #14477 from biglobster/keliang.

(cherry picked from commit 1275f64)
Signed-off-by: Sean Owen <sowen@cloudera.com>
@asfgit asfgit closed this in 1275f64 Aug 7, 2016
@srowen
Copy link
Member

srowen commented Aug 7, 2016

Merged to master/2.0

@liancheng
Copy link
Contributor

@srowen Sorry for the late reply, this LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants