[SPARK-33363] Add prompt information related to the current task when pyspark/sparkR starts #30266

echohlne · 2020-11-05T17:20:00Z

What changes were proposed in this pull request?

add prompt information about current applicationId, current URL and master info when pyspark / sparkR starts.

Why are the changes needed?

The information printed when pyspark/sparkR starts does not prompt the basic information of current application, and it is not convenient when used pyspark/sparkR in dos.

Does this PR introduce any user-facing change?

no

How was this patch tested?

manual test result shows below:

HyukjinKwon · 2020-11-06T03:48:40Z

ok to test

python/pyspark/shell.py

SparkQA · 2020-11-06T04:28:50Z

Test build #130684 has finished for PR 30266 at commit d780999.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-06T04:34:15Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35295/

SparkQA · 2020-11-06T04:55:28Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35295/

SparkQA · 2020-11-06T09:21:56Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35316/

SparkQA · 2020-11-06T09:34:24Z

Test build #130706 has finished for PR 30266 at commit a8bb388.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-06T09:53:39Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35316/

SparkQA · 2020-11-06T10:53:57Z

Test build #130716 has finished for PR 30266 at commit aae1264.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-06T10:55:59Z

Test build #130715 has finished for PR 30266 at commit a929e18.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-06T11:09:00Z

Test build #130718 has finished for PR 30266 at commit 0aaddd2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-06T11:15:46Z

Test build #130719 has finished for PR 30266 at commit 762b86e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-06T11:43:48Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35325/

SparkQA · 2020-11-06T11:48:15Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35327/

SparkQA · 2020-11-06T11:58:11Z

Test build #130721 has finished for PR 30266 at commit 976e2ce.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-06T12:07:33Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35325/

SparkQA · 2020-11-06T12:09:20Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35327/

SparkQA · 2020-11-06T12:30:07Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35328/

SparkQA · 2020-11-06T12:52:04Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35328/

SparkQA · 2020-11-06T12:52:39Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35330/

SparkQA · 2020-11-06T13:18:28Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35330/

echohlne · 2020-11-08T14:39:35Z

@HyukjinKwon hi,please help me to review this when you have time, thanks~.

HyukjinKwon · 2020-11-09T02:21:55Z

R/pkg/inst/profile/shell.R

@@ -43,5 +43,8 @@
  cat("      /_/", "\n")
  cat("\n")

+  cat("\nSpark context Web UI available at", SparkR::sparkR.uiWebUrl())
+  cat("\nSpark context available as 'sc' (master = ", unlist(SparkR::sparkR.conf("spark.master")),


Let's remove this line. Spark context is a private API in SparkR.

Namely something like:

cat("\nSparkSession available as 'spark' (master = ", unlist(SparkR::sparkR.conf("spark.master")), ", app id = ", unlist(SparkR::sparkR.conf("spark.app.id")), ").", sep = "")

HyukjinKwon

Looks good otherwise.

SparkQA · 2020-11-09T13:01:08Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35394/

SparkQA · 2020-11-09T13:17:15Z

Test build #130785 has finished for PR 30266 at commit 732cb2a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-09T13:21:47Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35394/

SparkQA · 2020-11-09T15:29:28Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35402/

SparkQA · 2020-11-09T15:35:06Z

Test build #130793 has finished for PR 30266 at commit 9742ad9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-09T15:52:06Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35402/

echohlne · 2020-11-09T16:14:41Z

@HyukjinKwon done this, please help to review it again, thanks a lot.

HyukjinKwon · 2020-11-10T02:12:01Z

Merged to master.

Update shell.py

c03d386

github-actions bot added CORE PYTHON labels Nov 5, 2020

Update shell.py

d780999

HyukjinKwon reviewed Nov 6, 2020

View reviewed changes

python/pyspark/shell.py Show resolved Hide resolved

Update shell.R

a8bb388

github-actions bot added the R label Nov 6, 2020

echohlne changed the title ~~[SPARK-33363] Add prompt information related to the current task when pyspark starts~~ [SPARK-33363] Add prompt information related to the current task when pyspark/sparkR starts Nov 6, 2020

echohlne added 3 commits November 6, 2020 17:42

Update shell.R

a929e18

Update shell.R

d093da9

Update shell.R

aae1264

echohlne added 2 commits November 6, 2020 18:10

Update shell.R

0aaddd2

Update shell.R

762b86e

Update shell.R

976e2ce

HyukjinKwon reviewed Nov 9, 2020

View reviewed changes

HyukjinKwon approved these changes Nov 9, 2020

View reviewed changes

Update shell.R

732cb2a

Update shell.R

9742ad9

HyukjinKwon closed this in 4360c6f Nov 10, 2020

echohlne deleted the pyspark-hint-info branch November 10, 2020 02:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-33363] Add prompt information related to the current task when pyspark/sparkR starts #30266

[SPARK-33363] Add prompt information related to the current task when pyspark/sparkR starts #30266

echohlne commented Nov 5, 2020 •

edited

HyukjinKwon commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

echohlne commented Nov 8, 2020

HyukjinKwon Nov 9, 2020

HyukjinKwon Nov 9, 2020

HyukjinKwon left a comment

SparkQA commented Nov 9, 2020

SparkQA commented Nov 9, 2020

SparkQA commented Nov 9, 2020

SparkQA commented Nov 9, 2020

SparkQA commented Nov 9, 2020

SparkQA commented Nov 9, 2020

echohlne commented Nov 9, 2020

HyukjinKwon commented Nov 10, 2020

[SPARK-33363] Add prompt information related to the current task when pyspark/sparkR starts #30266

[SPARK-33363] Add prompt information related to the current task when pyspark/sparkR starts #30266

Conversation

echohlne commented Nov 5, 2020 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

HyukjinKwon commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

echohlne commented Nov 8, 2020

HyukjinKwon Nov 9, 2020

Choose a reason for hiding this comment

HyukjinKwon Nov 9, 2020

Choose a reason for hiding this comment

HyukjinKwon left a comment

Choose a reason for hiding this comment

SparkQA commented Nov 9, 2020

SparkQA commented Nov 9, 2020

SparkQA commented Nov 9, 2020

SparkQA commented Nov 9, 2020

SparkQA commented Nov 9, 2020

SparkQA commented Nov 9, 2020

echohlne commented Nov 9, 2020

HyukjinKwon commented Nov 10, 2020

echohlne commented Nov 5, 2020 •

edited