[V2][Ray] Make KubeRay submission mode configurable - #7799
Conversation
Signed-off-by: davidlin20dev <davidlin20.dev@gmail.com>
Signed-off-by: davidlin20dev <davidlin20.dev@gmail.com>
| option go_package = "github.com/flyteorg/flyte/v2/gen/go/flyteidl2/plugins"; | ||
|
|
||
| // SubmissionMode specifies how the KubeRay operator submits the Ray job to the RayCluster. | ||
| enum SubmissionMode { |
There was a problem hiding this comment.
I think we just need to update the ray config here.
flyte/flyteplugins/go/tasks/plugins/k8s/ray/config.go
Lines 61 to 63 in 1dd0464
The submission mode should be decided by the platform team
There was a problem hiding this comment.
@pingsutw Thanks for the review! Just want to make sure I'm on the right track before I continue.
@popojk mentioned in Slack that users should be able to specify the mode, with both SDK and backend changes (which is what this PR implements), so I assumed we'd let users override the mode from the SDK. And what you mean is that only the platform team should pick the setting, is that correct?
So, should I do config-only, per-task only, or both (config default + per-task override, like ShutdownAfterJobFinishes)?
There was a problem hiding this comment.
yup, ray plugin config is usually set by the platform team, so we only need to do config-only
There was a problem hiding this comment.
We can add it to the proto in the future if someone really wants to override it per task
There was a problem hiding this comment.
I see, thanks for the clarification!
According to Kuberay doc, submitter also support sidecar mode, which means the ray operator will inject submitter into header node as sidecar. @pingsutw Do you think we should also support it as the config settings seems easy? |
Per review feedback, the submission mode is a platform-level concern: drop the SubmissionMode proto field and read the mode from the Ray plugin config instead. Invalid config values fail task construction. Signed-off-by: davidlin20dev <davidlin20.dev@gmail.com>
|
yup, let's add that too |
Per review discussion, also accept KubeRay's SidecarMode, which injects the submitter as a container in the head pod. Flyte already defaults Ray pod restartPolicy to Never, which SidecarMode requires. Signed-off-by: davidlin20dev <davidlin20.dev@gmail.com>
| case string(rayv1.K8sJobMode), "": | ||
| // submissionMode already defaults to K8sJobMode | ||
| default: | ||
| return nil, fmt.Errorf("invalid ray submission mode %q: must be K8sJobMode, HTTPMode or SidecarMode", cfg.SubmissionMode) |
There was a problem hiding this comment.
We could return flyteerr.Errorf(flyteerr.BadTaskSpecification, ), so the task won't be retried on this error
There was a problem hiding this comment.
Done. Thanks for the pointer! One question, I'm curious: I traced the code and understand flyteerr classifies the error with a code, but I couldn't find where that code prevents retries. It looks like plugin errors all go through recordSystemError and get retried regardless of code. Am I missing something?
Matches the file's existing error convention and prevents retries on a deterministic configuration error. Signed-off-by: davidlin20dev <davidlin20.dev@gmail.com>
70b1206 to
fb07f04
Compare
3 participants
What changes were proposed in this pull request?
Adds a
submissionModesetting to the Ray plugin configuration, letting platform operators choose how the KubeRay operator submits Ray jobs:K8sJobMode(default): a submitter Kubernetes Job runsray job submitagainst the RayCluster. This is the existing behavior and remains the default.HTTPMode: the KubeRay operator submits the job via an HTTP request directly to the Ray head node. No submitter pod is created, so job failures caused by submitter pod eviction are eliminated.SidecarMode: the KubeRay operator injects the submitter as a sidecar container in the Ray head pod. The submitter shares the head pod's fate, so it cannot be evicted independently of the job. Requires the head pod'srestartPolicyto beNever, which Flyte already sets by default.In
HTTPModeandSidecarMode, the plugin skips building the submitter pod template, since no separate submitter pod exists. An invalidsubmissionModevalue fails task construction with an explicit error rather than silently falling back.Why are the changes needed?
In
K8sJobMode, the submitter pod can be evicted independently of the Ray cluster (e.g., during node drains), causing KubeRay to mark an otherwise healthy job as failed.HTTPModeandSidecarModeboth remove the standalone submitter pod from the picture. See #7772.Per review discussion, the submission mode is a platform-level reliability concern, so it is exposed via plugin configuration (decided by the platform team) rather than per-task API. A per-task proto override can be added in the future if needed.
How was this patch tested?
K8sJobMode, unset config (defaults toK8sJobMode),HTTPMode(asserts no submitter pod template is set),SidecarMode(asserts no submitter pod template is set), and an invalid value (asserts a hard error).config_flags.goregenerated with thepflagsgenerator.flyteplugins/.../k8s/raypackage test suite passes locally.Labels
Related to #7772