Op that runs a kubernetes job #8161
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ |
gibsondan
force-pushed
the
createk8sop
branch
3 times, most recently
from
9d38608
to
1f7db88
Compare
|
will add some more tests / termination support etc. but curious for any initial thoughts |
|
Sweet. One q is should it be an op or a resource (that basically just wraps our client methods). I think @OwenKephart had mentioned that leaning towards resources can be preferred? It simplifies unit testing I suppose |
|
Yeah I was talking with @sryza about this a bit yesterday too. I'm not totally opposed to pulling the k8s-specific part of this into a resource and making this a "run an arbitrary container op" that depends on a particular resource (almost like a steplauncher, sort of... ) but I also think this there might just be people out there looking for the same KubernetesPodOperator experience, but in Dagster? |
|
I don't think it should just be a resource though, in case that's what you're proposing. If we only provided the resource I think you'd end up with a lot of boilerplate every time you wanted to use it to run a container. |
|
late to the party, but I'm not concerned about the lack of resource-y ness for this particular integration. There are basically two reasons that I tend to push lots of implementation details into resources when writing integrations:
Because it seems like most of the config here is unique per-op, and it's possible to run k8s locally, this setup seems ok to me. |
johannkm
approved these changes
Jun 9, 2022
gibsondan
force-pushed
the
createk8sop
branch
2 times, most recently
from
350731d
to
e351d3b
Compare
gibsondan
changed the title
Summary: This introduces a more direct analogue to a 'pod that manages a k8s container' - to handle cases where you want to stitch together a mixture of ops, or orchestrate non-Python languages. A nice thing is that we can use container_context to still pass config through to ops like this (so you can set up a default set of secrets to include in all ops even when you're not launching the run in k8s, for example). And we can expose raw job_config on the op as well to give you access to the full k8s api for the job.
prha
pushed a commit
that referenced
this pull request
Jul 6, 2022
* WIP: Op that runs a kubernetes job Summary: This introduces a more direct analogue to a 'pod that manages a k8s container' - to handle cases where you want to stitch together a mixture of ops, or orchestrate non-Python languages. A nice thing is that we can use container_context to still pass config through to ops like this (so you can set up a default set of secrets to include in all ops even when you're not launching the run in k8s, for example). And we can expose raw job_config on the op as well to give you access to the full k8s api for the job. * Make wait_for_pod disable timeout on 0 Co-authored-by: Johann Miller <johann@elementl.com>
Jiafi
pushed a commit
to Jiafi/dagster
that referenced
this pull request
Jul 11, 2022
* WIP: Op that runs a kubernetes job Summary: This introduces a more direct analogue to a 'pod that manages a k8s container' - to handle cases where you want to stitch together a mixture of ops, or orchestrate non-Python languages. A nice thing is that we can use container_context to still pass config through to ops like this (so you can set up a default set of secrets to include in all ops even when you're not launching the run in k8s, for example). And we can expose raw job_config on the op as well to give you access to the full k8s api for the job. * Make wait_for_pod disable timeout on 0 Co-authored-by: Johann Miller <johann@elementl.com>
gibsondan
added a commit
that referenced
this pull request
Jul 26, 2022
Summary: Overriding the command in container_config changed in #8161 - we override the passed in command to the function, even if it is empty. Instead, fall back to the user-provided container_config if it is set. Test Plan: BK
gibsondan
added a commit
that referenced
this pull request
Jul 26, 2022
Summary: Overriding the command in container_config changed in #8161 - we override the passed in command to the function, even if it is empty. Instead, fall back to the user-provided container_config if it is set. Test Plan: BK
gibsondan
added a commit
that referenced
this pull request
Jul 26, 2022
Summary: Overriding the command in container_config changed in #8161 - we override the passed in command to the function, even if it is empty. Instead, fall back to the user-provided container_config if it is set. Test Plan: BK
gibsondan
added a commit
that referenced
this pull request
Jul 26, 2022
Summary: Overriding the command in container_config changed in #8161 - we override the passed in command to the function, even if it is empty. Instead, fall back to the user-provided container_config if it is set. Test Plan: BK
gibsondan
added a commit
that referenced
this pull request
Jul 26, 2022
Summary: Overriding the command in container_config changed in #8161 - we override the passed in command to the function, even if it is empty. Instead, fall back to the user-provided container_config if it is set. Test Plan: BK
|
Yeah I’m ok with just an op for now
…On Thu, Jun 2, 2022 at 9:50 PM gibsondan ***@***.***> wrote:
I don't think it should *just* be a resource though, in case that's what
you're proposing. If we only provided the resource I think you'd end up
with a lot of boilerplate every time you wanted to use it to run a
container.
—
Reply to this email directly, view it on GitHub
<#8161 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFH7XHP336UYUOJO3A6YKGDVNFQHLANCNFSM5XTT65XQ>
.
You are receiving this because your review was requested.Message ID:
***@***.***>
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
This introduces a more direct analogue to a 'pod that manages a k8s container' - to handle cases where you want to stitch together a mixture of ops, or orchestrate non-Python languages.
A nice thing is that we can use container_context to still pass config through to ops like this (so you can set up a default set of secrets to include in all ops even when you're not launching the run in k8s, for example). And we can expose raw job_config on the op as well to give you access to the full k8s api for the job.
Summary & Motivation
How I Tested These Changes