Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(chart): Add baseline Hadoop w/Hive chart #6688

Closed
wants to merge 3 commits into from

Conversation

@sylus
Copy link

commented Jul 17, 2018

What this PR does / why we need it: Placeholder Hadoop w/Hive chart (https://hive.apache.org/)

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): N/A

Special notes for your reviewer: This P.R temporarily copies the Hadoop chart with Hive installed alongside as a SFS with PVC etc (about 5 files added in total). I wasn't sure if this could just be an optional add-on to the stable Helm chart so for now just pushed all of this to incubator. This was primarily needed for me to get the Kylo chart in a different P.R. #5773 fully working. The values.yaml was also tweaked to run optimally on a single node alongside the expected Kylo.

In the incubator/hadoop folder:

helm install --namespace hadoop --name hadoop -f values.yaml .

Tasks

  • Properly extend Hadoop chart via requirements.yaml
  • Add HCatalog
  • Specific docs for each scenario

Relevant Links

https://github.com/govcloud/docker-hadoop
https://hub.docker.com/r/govcloud/docker-hadoop/builds/

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Jul 17, 2018

Hi @sylus. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot requested review from linki and sameersbn Jul 17, 2018

@sylus sylus referenced this pull request Jul 17, 2018
2 of 2 tasks complete
@shawnzhu
Copy link
Contributor

left a comment

It looks like the code is from chart stable/hadoop but committed under your account.
I would like to see a dedicated Hive chart with:

  • HCatalog (with embedded derby or RDBMS as backend)
  • Hive server
  • (optional) HDFS cluster from the chart stable/hadoop managed by requirements.yaml.

So that it can support the scenario that running a standalone Hive server without HDFS (a.k.a., uses local warehouse).

@sylus

This comment has been minimized.

Copy link
Author

commented Jul 18, 2018

@shawnzhu Yes as stated in my description this is simply the hadoop chart extended with the hive SFS and other components (about 5 files) apologies if didn't make that clear. I just quickly needed these adjustments to get the Kylo chart I submitted a P.R. for working so this was def just a stopgap till knew the proper direction.

I wasn't sure whether I should just add the hive logic straight to the hadoop chart in stable via P.R. (whether would be accepted as might not want to complicate it) and have it just be a flag or make a whole new project with hive namespace and extending hadoop like you mentioned via requirements.yaml.

I can definitely do the last two steps and will properly extend the hadoop chart if that is the correct direction that makes the most sense.

I added your directions as tasks in the top for me to action which should be done in a few days or so ^_^

Thanks!

@sylus sylus force-pushed the govcloud:chart-incubator-hadoop branch Jul 18, 2018

@k8s-ci-robot k8s-ci-robot added size/XL and removed size/XXL labels Jul 18, 2018

@sylus

This comment has been minimized.

Copy link
Author

commented Jul 18, 2018

@shawnzhu I temporarily just updated the P.R. so just shows the delta's added to Hadoop chart rather then copy to incubator.

Will still proceed the route you suggested but will need to also submit a few P.R. to Hadoop to successfully extend. Override their configmap settings for some files, and add WASB support.

<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:postgresql://hadoop-postgresql:5432/metastore?createDatabaseIfNotExist=true</value>

This comment has been minimized.

Copy link
@shawnzhu

shawnzhu Aug 30, 2018

Contributor

it doesn't need createDatabaseIfNotExist=true since it will create the db metastore when provisioning postgresql (you've specified postgresDatabase: metastore in values.yaml)

@lalarsson

This comment has been minimized.

Copy link

commented Sep 11, 2018

@shawnzhu @sylus I'd love to use this for a project at work. Would it be ok if I helped out with the last details?

@sylus

This comment has been minimized.

Copy link
Author

commented Sep 17, 2018

@lalarsson that would be amazing :)

@stale

This comment has been minimized.

Copy link

commented Oct 17, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

@stale stale bot added the lifecycle/stale label Oct 17, 2018

@sylus sylus referenced this pull request Oct 19, 2018
0 of 2 tasks complete
@sylus

This comment has been minimized.

Copy link
Author

commented Oct 19, 2018

/remove-lifecycle stale

@stale stale bot removed the lifecycle/stale label Oct 19, 2018

sylus added some commits Oct 19, 2018

feat(chart): Add baseline Hadoop w/Hive chart
Signed-off-by: William Hearn <sylus1984@gmail.com>
feat(chart): Add baseline Hadoop w/Hive chart
Signed-off-by: William Hearn <sylus1984@gmail.com>

@sylus sylus force-pushed the govcloud:chart-incubator-hadoop branch to 27ac933 Oct 19, 2018

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Oct 19, 2018

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sylus
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: linki

If they are not already assigned, you can assign the PR to them by writing /assign @linki in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

feat(chart): Add baseline Hadoop w/Hive chart
Signed-off-by: William Hearn <sylus1984@gmail.com>
@stale

This comment has been minimized.

Copy link

commented Nov 18, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

@stale stale bot added the lifecycle/stale label Nov 18, 2018

@sylus

This comment has been minimized.

Copy link
Author

commented Nov 18, 2018

/remove-lifecycle stale

@stale stale bot removed the lifecycle/stale label Nov 18, 2018

@stale

This comment has been minimized.

Copy link

commented Dec 18, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

@stale stale bot added the lifecycle/stale label Dec 18, 2018

@sylus

This comment has been minimized.

Copy link
Author

commented Dec 19, 2018

/remove-lifecycle stale

@stale stale bot removed the lifecycle/stale label Dec 19, 2018

@stale

This comment has been minimized.

Copy link

commented Jan 18, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

@stale stale bot added the lifecycle/stale label Jan 18, 2019

@sylus

This comment has been minimized.

Copy link
Author

commented Jan 19, 2019

/remove-lifecycle stale

@stale stale bot removed the lifecycle/stale label Jan 19, 2019

@stale

This comment has been minimized.

Copy link

commented Feb 18, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

@stale stale bot added the lifecycle/stale label Feb 18, 2019

@sylus

This comment has been minimized.

Copy link
Author

commented Feb 18, 2019

/remove-lifecycle stale

@stale stale bot removed the lifecycle/stale label Feb 18, 2019

@stale

This comment has been minimized.

Copy link

commented Mar 20, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

@stale stale bot added the lifecycle/stale label Mar 20, 2019

@sylus

This comment has been minimized.

Copy link
Author

commented Mar 21, 2019

/remove-lifecycle stale

@stale stale bot removed the lifecycle/stale label Mar 21, 2019

@chino

This comment has been minimized.

Copy link

commented Apr 16, 2019

Is this still being actively worked on?

@stale

This comment has been minimized.

Copy link

commented May 16, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

@stale stale bot added the lifecycle/stale label May 16, 2019

@maorfr

This comment has been minimized.

Copy link
Member

commented May 16, 2019

hey @sylus

Have you considered hosting this chart and adding it to helm hub?
https://github.com/helm/hub/blob/master/Repositories.md

@stale stale bot removed the lifecycle/stale label May 16, 2019

@stale

This comment has been minimized.

Copy link

commented Jun 15, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

@stale stale bot added the lifecycle/stale label Jun 15, 2019

@stale

This comment has been minimized.

Copy link

commented Jun 29, 2019

This issue is being automatically closed due to inactivity.

@stale stale bot closed this Jun 29, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.