Skip to content

[MINOR][DOCS] Mention other Python dependency tools in documentation#34134

Closed
HyukjinKwon wants to merge 2 commits intoapache:masterfrom
HyukjinKwon:minor-docs-py-deps
Closed

[MINOR][DOCS] Mention other Python dependency tools in documentation#34134
HyukjinKwon wants to merge 2 commits intoapache:masterfrom
HyukjinKwon:minor-docs-py-deps

Conversation

@HyukjinKwon
Copy link
Member

@HyukjinKwon HyukjinKwon commented Sep 29, 2021

What changes were proposed in this pull request?

Self-contained.

Why are the changes needed?

For user's more information on available Python dependency management in PySpark.

Does this PR introduce any user-facing change?

Yes, documentation change.

How was this patch tested?

Manaully built the docs and checked the results:
Screen Shot 2021-09-29 at 10 11 56 AM
Screen Shot 2021-09-29 at 10 12 22 AM
Screen Shot 2021-09-29 at 10 12 42 AM

@github-actions github-actions bot added the DOCS label Sep 29, 2021
@SparkQA
Copy link

SparkQA commented Sep 29, 2021

Test build #143691 has finished for PR 34134 at commit 791fa7e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 29, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48205/

mkgray
mkgray suggested changes Sep 29, 2021
View reviewed changes
docs/rdd-programming-guide.md Outdated
variable called `sc`. Making your own SparkContext will not work. You can set which master the
context connects to using the `--master` argument, and you can add Python .zip, .egg or .py files
to the runtime path by passing a comma-separated list to `--py-files`. You can also add dependencies
to the runtime path by passing a comma-separated list to `--py-files`. For thrid-party Python dependencies,
Copy link

@mkgray mkgray Sep 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor typo, should be third-party

Copy link
Member Author

@HyukjinKwon HyukjinKwon Sep 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, thanks man

docs/submitting-applications.md Outdated
For Python, you can use the `--py-files` argument of `spark-submit` to add `.py`, `.zip` or `.egg`
files to be distributed with your application. If you depend on multiple Python files we recommend
packaging them into a `.zip` or `.egg`.
packaging them into a `.zip` or `.egg`. For thrid-party Python dependencies,
Copy link

@mkgray mkgray Sep 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor typo, should be third-party

@SparkQA
Copy link

SparkQA commented Sep 29, 2021

Test build #143697 has finished for PR 34134 at commit 6e31a31.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 29, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48205/

@SparkQA
Copy link

SparkQA commented Sep 29, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48213/

@SparkQA
Copy link

SparkQA commented Sep 29, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48213/

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Sep 29, 2021

Thanks guys.

Merged to master. Let me merge this in branch-3.2 too - very trivial doc change.

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Sep 29, 2021

Merged to master and branch-3.2.

HyukjinKwon added a commit that referenced this pull request Sep 29, 2021
@HyukjinKwon
[MINOR][DOCS] Mention other Python dependency tools in documentation
939c4d9 
### What changes were proposed in this pull request?

Self-contained.

### Why are the changes needed?

For user's more information on available Python dependency management in PySpark.

### Does this PR introduce _any_ user-facing change?
Yes, documentation change.

### How was this patch tested?
Manaully built the docs and checked the results:
<img width="918" alt="Screen Shot 2021-09-29 at 10 11 56 AM" src="https://user-images.githubusercontent.com/6477701/135186536-2f271378-d06b-4c6b-a4be-691ce395db9f.png">
<img width="976" alt="Screen Shot 2021-09-29 at 10 12 22 AM" src="https://user-images.githubusercontent.com/6477701/135186541-0f4c5615-bc49-48e2-affd-dc2f5c0334bf.png">
<img width="920" alt="Screen Shot 2021-09-29 at 10 12 42 AM" src="https://user-images.githubusercontent.com/6477701/135186551-0b613096-7c86-4562-b345-ddd60208367b.png">

Closes #34134 from HyukjinKwon/minor-docs-py-deps.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 13c2b71)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Sep 29, 2021

+1, late LGTM.

sunchao pushed a commit to sunchao/spark that referenced this pull request Dec 8, 2021
@HyukjinKwon @dongjoon-hyun
[MINOR][DOCS] Mention other Python dependency tools in documentation
0a8f3b8 
### What changes were proposed in this pull request?

Self-contained.

### Why are the changes needed?

For user's more information on available Python dependency management in PySpark.

### Does this PR introduce _any_ user-facing change?
Yes, documentation change.

### How was this patch tested?
Manaully built the docs and checked the results:
<img width="918" alt="Screen Shot 2021-09-29 at 10 11 56 AM" src="https://user-images.githubusercontent.com/6477701/135186536-2f271378-d06b-4c6b-a4be-691ce395db9f.png">
<img width="976" alt="Screen Shot 2021-09-29 at 10 12 22 AM" src="https://user-images.githubusercontent.com/6477701/135186541-0f4c5615-bc49-48e2-affd-dc2f5c0334bf.png">
<img width="920" alt="Screen Shot 2021-09-29 at 10 12 42 AM" src="https://user-images.githubusercontent.com/6477701/135186551-0b613096-7c86-4562-b345-ddd60208367b.png">

Closes apache#34134 from HyukjinKwon/minor-docs-py-deps.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 13c2b71)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
@HyukjinKwon HyukjinKwon deleted the minor-docs-py-deps branch January 4, 2022 00:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@viirya viirya viirya approved these changes

@ueshin ueshin ueshin approved these changes

+1 more reviewer

@mkgray mkgray mkgray requested changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

DOCS

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@HyukjinKwon @SparkQA @dongjoon-hyun @viirya @ueshin @mkgray