You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add optional resource limits (cap concurrent batches / estimated memory) and configuration validation (error on invalid input instead of silently rewriting it).
Motivation
Under heavy load there's currently nothing preventing unbounded concurrent batches or memory growth. Separately, invalid config is silently "fixed" rather than reported (see the ROAST.md hardening issue, point #4) — e.g. MinItems: 0 is silently rewritten to 1 in fixConfig (batch/batch.go:453 on master), which leads to surprising behavior.
Proposed API
From feat/resource-limits-validation @ bfa33e2:
typeResourceLimitsstruct { /* max active batches, max estimated memory, ... */ }
funcDefaultResourceLimits() ResourceLimits// sensible defaults from system resourcesfunc (rResourceLimits) Validate() errortypeOptionsstruct {
Config batch.ConfigResourceLimits*ResourceLimits
}
func (o*Options) WithDefaults() *OptionsfuncNewWithOptions(opts*Options) *Batch// recommended constructor when using limits// Examplelimits:=batch.DefaultResourceLimits()
b:=batch.NewWithOptions(&batch.Options{
Config: batch.NewConstantConfig(&batch.ConfigValues{MinItems: 10, MaxItems: 100}),
ResourceLimits: &limits,
})
Internally the prototype used a resourceTracker (canStartBatch/startBatch/finishBatch/getUsage) to enforce limits and a Validate() pass on config.
Design notes
Two related concerns bundled here — could split into "resource limits" and "config validation" if preferred.
Config validation should resolve ROAST Add more examples to readme #4: prefer returning an error over silent mutation (decide per field; some defaulting is fine, but MinItems: 0 → error or documented default).
Summary
Add optional resource limits (cap concurrent batches / estimated memory) and configuration validation (error on invalid input instead of silently rewriting it).
Motivation
Under heavy load there's currently nothing preventing unbounded concurrent batches or memory growth. Separately, invalid config is silently "fixed" rather than reported (see the ROAST.md hardening issue, point #4) — e.g.
MinItems: 0is silently rewritten to1infixConfig(batch/batch.go:453onmaster), which leads to surprising behavior.Proposed API
From
feat/resource-limits-validation@bfa33e2:Internally the prototype used a
resourceTracker(canStartBatch/startBatch/finishBatch/getUsage) to enforce limits and aValidate()pass on config.Design notes
MinItems: 0→ error or documented default).Batch[T]/NewWithOptions[T].Source (for recovery — branch is being deleted)
feat/resource-limits-validation@bfa33e2— files:batch/resource_limits.go,batch/options.go,batch/config.go,batch/config_validation_test.go,batch/resource_limits_test.go. (PR #46 was closed; its diff remains viewable on the PR.)Related: #75 (config-validation / silent-mutation, ROAST point #4)