-
Notifications
You must be signed in to change notification settings - Fork 42
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[LIMINAL-56] add default executor (k8s) for spark #59
[LIMINAL-56] add default executor (k8s) for spark #59
Conversation
f72ec63
to
339bce4
Compare
@@ -0,0 +1,17 @@ | |||
# |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@assapin you can start with the k8s example
(i will add readme soon, meanwhile, you can use the getting_started doc with examples/spark-app-demo/k8s files)
examples/spark-app-demo/emr/configs/emr_cloudformation_template.yml
Outdated
Show resolved
Hide resolved
9393db1
to
8befe38
Compare
59f7ad3
to
af13d1c
Compare
spark: | ||
executor: k8s | ||
image: my_spark_image | ||
source: . |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need to create images section
@@ -39,6 +39,7 @@ pipeline_defaults: | |||
type: python | |||
description: static input task | |||
image: my_static_input_task_image | |||
no_cache: True |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
need to add images section
@@ -43,6 +43,7 @@ pipelines: | |||
type: python | |||
description: parallelized static input task | |||
image: my_static_input_task_image | |||
no_cache: True |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
need to add images section
No description provided.