feat: add --duration flag for self-terminated load runs#31
Merged
Conversation
Wraps the load-test ctx in context.WithTimeout when --duration > 0. When the timeout fires, the existing graceful-shutdown path runs (background tasks unwind via ctx.Done(), final stats emit, post- summary flush completes), and the existing errors.Is check at exit boundary treats DeadlineExceeded as exit 0. Motivation: K8s Job-level activeDeadlineSeconds always sets the Job condition to Failed/DeadlineExceeded regardless of container exit code. With --duration set inside the deadline, seiload exits cleanly before K8s force-terminates, so the Job goes Complete. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
amir-deris
approved these changes
Apr 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
--durationflag so seiload self-terminates cleanly inside K8s JobactiveDeadlineSeconds, producing pod exit 0 → Job conditionCompleteinstead of K8s-mandatedFailed/DeadlineExceeded.Why this exists
Followup to #30. The exit-code fix in #30 is correct in isolation (graceful SIGTERM → exit 0), but on Kubernetes Jobs with Job-level
activeDeadlineSeconds, the K8s Job controller setscondition=Failed, reason=DeadlineExceededregardless of the container's exit code:A run on harbor's
nightlynamespace today (workflow 25189102396) confirmed this: seiload's pod was SIGTERMed at the deadline and likely exited 0 (post-#30), but the Job condition was set:This means
kube_job_failed=1andKubeJobFailedkeeps firing — the original symptom that motivated this whole investigation. The actual fix has to make seiload self-terminate before K8s decides "DeadlineExceeded."What changes
When
--durationis set:WithTimeout.durationelapses, ctx is canceled withcontext.DeadlineExceeded.ctx.Done().service.Runreturns the wrapped DeadlineExceeded error.EmitRunSummaryruns, post-summary flush sleeps for 45s.errors.Is(err, context.DeadlineExceeded)) clears the error.The existing post-summary flush delay still runs by design — it sits after
service.Runreturns, in the cleanup pipeline.What doesn't change
0(unlimited) so existing callers without the flag are unaffected.main.go:347-352is untouched and still works.Test plan
GOWORK=off go build .passesGOWORK=off go vet ./...clean--duration=${DURATION_MINUTES}mto seiload args; tomorrow's nightly will verify Job condition flips toCompleteCompanion change
Will follow with a small PR on
sei-protocol/platformthat:--duration=${DURATION_MINUTES}mto seiload args inclusters/harbor/nightly/templates/seiload-job.yaml.JOB_DEADLINE_SECONDSslightly to keepactiveDeadlineSecondsas a backstop only.🤖 Generated with Claude Code