-
Notifications
You must be signed in to change notification settings - Fork 173
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feat/emr eks spark operator #220
Feat/emr eks spark operator #220
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lmouhib Thanks for raising the PR for this new feature. We don't need a new blueprint for this pattern. We can use the existing emr-eks-karpenter
blueprint and introduce a new variable for deploying emr-spark-operator add-on.
- Add a new variable to deploy
emr-spark-operator
add-on - Add this new add-on under
workshop/modules/terraform-aws-eks-data-addons
- Add new example under
analytics/terraform/emr-eks-karpenter/examples/emr-spark-operator
andemr-spark-submit
etc. - You can have a dedicated Website doc similar to what you did in the PR
5c3d66c
to
4698724
Compare
@vara-bonthu I applied the changes requested |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Few minor comments otherwise looks good to me 馃憤
)) | ||
} | ||
|
||
resource "helm_release" "emr_spark_operator" { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You need to use all the values that comes with helm release. Use the other helm release example.
cpu: 100m | ||
memory: 512Mi | ||
|
||
# batchScheduler: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove the commented lines
# -- Webhook service port | ||
port: 8080 | ||
|
||
emrContainers: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Set only this value in this module and ignore the other values as those are the default
@@ -31,6 +31,10 @@ locals { | |||
grafana_repository = "https://grafana.github.io/helm-charts" | |||
grafana_version = "6.52.4" | |||
|
|||
emr_spark_operator_name = "spark-operator" | |||
emr_spark_operator_version = "1.1.26-amzn-1" | |||
emr_spark_operator_namespace = "spark-operator" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
change this to emr-spark-operator
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lmouhib Thanks for this PR 馃殌
What does this PR do?
Add a blueprint to use Spark Operator with EMR on EKS.
馃洃 Please open an issue first to discuss any significant work and flesh out details/direction - we would hate for your time to be wasted.
Consult the CONTRIBUTING guide for submitting pull-requests.
Motivation
Recently launched feature by emr on eks
More
website/docs
orwebsite/blog
section for this featurepre-commit run -a
with this PR. Link for installing pre-commit locallyFor Moderators
Additional Notes