Skip to content

Resource limits + configuration validation #73

Description

@MasterOfBinary

Summary

Add optional resource limits (cap concurrent batches / estimated memory) and configuration validation (error on invalid input instead of silently rewriting it).

Motivation

Under heavy load there's currently nothing preventing unbounded concurrent batches or memory growth. Separately, invalid config is silently "fixed" rather than reported (see the ROAST.md hardening issue, point #4) — e.g. MinItems: 0 is silently rewritten to 1 in fixConfig (batch/batch.go:453 on master), which leads to surprising behavior.

Proposed API

From feat/resource-limits-validation @ bfa33e2:

type ResourceLimits struct { /* max active batches, max estimated memory, ... */ }
func DefaultResourceLimits() ResourceLimits          // sensible defaults from system resources
func (r ResourceLimits) Validate() error

type Options struct {
    Config         batch.Config
    ResourceLimits *ResourceLimits
}
func (o *Options) WithDefaults() *Options
func NewWithOptions(opts *Options) *Batch            // recommended constructor when using limits

// Example
limits := batch.DefaultResourceLimits()
b := batch.NewWithOptions(&batch.Options{
    Config:         batch.NewConstantConfig(&batch.ConfigValues{MinItems: 10, MaxItems: 100}),
    ResourceLimits: &limits,
})

Internally the prototype used a resourceTracker (canStartBatch/startBatch/finishBatch/getUsage) to enforce limits and a Validate() pass on config.

Design notes

  • Two related concerns bundled here — could split into "resource limits" and "config validation" if preferred.
  • Config validation should resolve ROAST Add more examples to readme #4: prefer returning an error over silent mutation (decide per field; some defaulting is fine, but MinItems: 0 → error or documented default).
  • Predates the feat: migrate GoBatch API to generics #60 generics migration; align with Batch[T]/NewWithOptions[T].

Source (for recovery — branch is being deleted)

feat/resource-limits-validation @ bfa33e2 — files: batch/resource_limits.go, batch/options.go, batch/config.go, batch/config_validation_test.go, batch/resource_limits_test.go. (PR #46 was closed; its diff remains viewable on the PR.)


Related: #75 (config-validation / silent-mutation, ROAST point #4)

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions