Skip to content
This repository has been archived by the owner on Feb 22, 2022. It is now read-only.

[stable/spark] Use spark 2.4.0 , hadoop 3.1.1 and spark-master pod for zeppelin notebook #11285

Closed
wants to merge 14 commits into from
Closed

[stable/spark] Use spark 2.4.0 , hadoop 3.1.1 and spark-master pod for zeppelin notebook #11285

wants to merge 14 commits into from

Conversation

dbanda
Copy link

@dbanda dbanda commented Feb 8, 2019 •
edited
Loading

What this PR does / why we need it:

This PR updates the docker images to use spark 2.4.0, hadoop 3.1.1, zeppelin 0.9.0 . It includes transparent build scripts in the docker files so that users can easily modify the build to use any version of spark/hadoop/zeppelin.

Which issue this PR fixes

(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

Updated Zeppelin to use spark 2.4.0/hadoop 3.1.1

This PR makes Zeppelin use the spark-master pod as its master by default. In previous releases, Zeppelin would submit jobs using the local[*] master URL

Special notes for your reviewer:

Checklist

[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

  • DCO signed
  • Chart Version bumped
  • Variables are documented in the README.md

@helm-bot helm-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Feb 8, 2019
@k8s-ci-robot
Copy link
Contributor

Hi @dbanda. Thanks for your PR.

I'm waiting for a helm member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Feb 8, 2019
@helm-bot helm-bot added Contribution Allowed If the contributor has signed the DCO or the CNCF CLA (prior to the move to a DCO). size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Feb 8, 2019
@helm-bot helm-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Feb 8, 2019
@dbanda dbanda changed the title Use spark 2.4.0 , hadoop 3.1.1 and spark-master pod for zeppelin notebook [stable/spark] Use spark 2.4.0 , hadoop 3.1.1 and spark-master pod for zeppelin notebook Feb 8, 2019
@helm-bot helm-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Feb 13, 2019
@dbanda
Copy link
Author

dbanda commented Feb 15, 2019

/assign @paulczar

@metrofun
Copy link

metrofun commented Feb 22, 2019 •
edited
Loading

Also waiting for Spark 2.4 helm out
Friendly ping @paulczar 😇

@helm-bot helm-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed Contribution Allowed If the contributor has signed the DCO or the CNCF CLA (prior to the move to a DCO). size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Feb 25, 2019
@helm-bot helm-bot added Contribution Allowed If the contributor has signed the DCO or the CNCF CLA (prior to the move to a DCO). size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Feb 25, 2019
@stale stale bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 7, 2019
@mhamilton723
Copy link

@lachie83 do you have any updates on what we should do next to keep this moving? Thanks a ton for your help throughout this process!

@stale stale bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 20, 2019
@sshakeri
Copy link

@dbanda glad I saw this pull request. The default chart installs spark 1.5 which won't work. Thanks for the updates

@ttimasdf
Copy link

ttimasdf commented Sep 2, 2019

For those who may interests, you can simply use charts from Microsoft official helm repo. I didn't go through the code but the README and maintainer is the same.

reference:
https://hub.helm.sh/charts/microsoft/spark
https://github.com/microsoft/charts

@sshakeri
Copy link

sshakeri commented Sep 4, 2019

Here is a simple version of this with Spark 2.4.3 & Jupyter https://github.com/sshakeri/Containerized_Spark

@adam-carbone
Copy link

I would be willing to help here as well.. I'm working on this exact same thing, would like to use the images created by spark build script though.

@stale
Copy link

stale bot commented Oct 17, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

@stale stale bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 17, 2019
@zanhsieh
Copy link
Collaborator

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 17, 2019
@stale stale bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 17, 2019
@dbanda
update images
ca0e1f3 
Signed-off-by: Dalitso Banda <dalitsohb@gmail.com>
@k8s-ci-robot k8s-ci-robot added the do-not-merge/invalid-owners-file Indicates that a PR should not merge because it has an invalid OWNERS file in it. label Oct 25, 2019
@dbanda
Copy link
Author

dbanda commented Oct 25, 2019

/retest

1 similar comment
@dbanda
Copy link
Author

dbanda commented Oct 30, 2019

/retest

@dbanda
update images
82b42e2 
Signed-off-by: Dalitso Banda <dalitsohb@gmail.com>
@zanhsieh
Copy link
Collaborator

zanhsieh commented Nov 1, 2019

/verify-owners

@k8s-ci-robot
Copy link
Contributor

The following users are mentioned in OWNERS file(s) but are not members of the helm org.

Once all users have been added as members of the org, you can trigger verification by writing /verify-owners in a comment.

  • dbanda
    • stable/spark/OWNERS
  • mhamilton723
    • stable/spark/OWNERS

@zanhsieh
Copy link
Collaborator

zanhsieh commented Nov 1, 2019

@dbanda
Adding owners / maintainers only helm members could do. Please make that a separate PR please, otherwise this PR won't pass e2e test. Thank you.

@stale
Copy link

stale bot commented Dec 1, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

@stale stale bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 1, 2019
@stale
Copy link

stale bot commented Dec 15, 2019

This issue is being automatically closed due to inactivity.

@stale stale bot closed this Dec 15, 2019
@Bennyweasl
Copy link

Bennyweasl commented Jun 4, 2020 •
edited
Loading

This should be reopened and merged. The only thing holding this up is the owner's file. @paulczar @scottrigby @mhamilton723 @dbanda @daixiang0

@daixiang0
Copy link
Collaborator

lgtm

@mhamilton723
Copy link

Hey @Bennyweasl any input needed from me for the ownersfile?

@Bennyweasl
Copy link

Bennyweasl commented Jul 28, 2020 •
edited
Loading

@mhamilton723

from:k8s-ci-robot
The following users are mentioned in OWNERS file(s) but are not members of the helm org.

Once all users have been added as members of the org, you can trigger verification by writing /verify-owners in a comment.

dbanda
stable/spark/OWNERS
mhamilton723
stable/spark/OWNERS

I don't think this can be merged until both owners are part of helm org

@dbanda
Copy link
Author

dbanda commented Jul 30, 2020

@Bennyweasl I updated the OWNERS file a while ago see: https://github.com/dbanda/charts/blob/master/stable/spark/OWNERS

For some reason the update is not reflected in PR. Is it because it is closed?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@scottrigby scottrigby scottrigby requested changes

@wirehead wirehead wirehead left review comments

@mhamilton723 mhamilton723 mhamilton723 left review comments

@lachie83 lachie83 lachie83 requested changes

@unguiculus unguiculus Awaiting requested review from unguiculus

@viglesiasce viglesiasce Awaiting requested review from viglesiasce

Assignees

@paulczar paulczar

Labels
Contribution Allowed If the contributor has signed the DCO or the CNCF CLA (prior to the move to a DCO). do-not-merge/invalid-owners-file Indicates that a PR should not merge because it has an invalid OWNERS file in it. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. ok-to-test size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Zeppelin does not use correct master with spark 2.4.0 image