Release 1.1: Batch Rework, Custom Individuals & Intervention Overhaul

[Julian-Patzner]

Welcome to the 1.1 release! This release rebuilds the batch API around a streaming, memory-bounded processing model, adds support for custom per-individual attributes, and overhauls the intervention engine for type-stable, allocation-free execution of measures.

Note that the batch rewrite and parts of the intervention internals contain breaking changes. Please read the relevant sections below before upgrading.

---

Batch Rework

The Batch workflow has been redesigned so that memory no longer scales with the number of runs. Instead of instantiating and retaining a Simulation (and later a ResultData) per run, a Batch now stores lightweight simulation configurations and folds each run's results into running accumulators as it goes.

• Streaming, memory-bounded processing: BatchProcessor now aggregates results using Welford online statistics (mean, variance, min/max, 95% CI). Each run is executed, accumulated, and discarded, so peak memory is proportional to a single run rather than to the size of the batch. A new WelfordState accumulator and helpers (welford_update!, welford_to_aggregate, welford_df_to_stats_df, welford_df_to_stats_df_multicol) back this model and produce the same aggregate schema as the existing aggregate_* utilities.
• Configuration-based Batch: Batch now holds a vector of simulation-config NamedTuples and per-run setup functions instead of Simulation objects. Create one with Batch(n_runs = N; simargs...), where simargs are any keyword arguments accepted by Simulation(). An optional setup = sim -> ... hook lets you attach interventions, strategies, or triggers to each run after construction but before it executes.
• process! replaces run!'s old semantics: process!(batch; ...) runs every configuration sequentially and returns a BatchProcessor. run!(::Batch) is retained as an alias for process! (it now returns a BatchProcessor rather than mutating and returning the Batch).
• Per-simulation seeding: Every run receives a deterministic seed derived from a single master seed. Pass seed = N to process!/BatchData for fully reproducible batches, and retrieve the seed actually used via seed(bp).
• Representative (median) run: Pass median_by = pp -> ... to select a criterion (e.g. total infections); the run whose criterion is closest to the median across the batch is re-run in full and stored as median_run(bp). For multi-group batches, one median run is computed per group. Disable with median_by = nothing (the default).
• Per-label / per-group aggregation: Pass group_by = :label (or any config field) to accumulate statistics separately per group, accessible via per_label(bd) / per_group. gemsplot(bd) then draws one mean±CI ribbon per label automatically.
• BatchData-native plots: gemsplot(bd::BatchData) now dispatches directly to mean±CI ribbon plots built from the streamed accumulators — no need to retain every per-run ResultData. BatchData(b) remains a convenient shorthand for BatchData(process!(b)).
• keep_rundata for individual runs: Individual run traces are retained by default in the lightweight LightRD format and are available via runs(bd). Pass keep_rundata = false to drop them entirely for minimum memory.
• customlogger passthrough: process!(batch; customlogger = ...) attaches an independent copy of a CustomLogger to each run, so logged data is never shared across runs.
• New accessors: BatchProcessor/BatchData gain cumulative_cases, dark_figure, generation_times, n_runs, r0, median_run, seed, and more.

Breaking changes

• Batch no longer stores Simulation objects. Use Batch(n_runs = N, kwarg = val, ...) and simconfigs(batch) / simsetups(batch).
• simulations(batch) has been removed; use simconfigs(batch).
• add!(::Simulation, ::Batch) and remove!(::Simulation, ::Batch) have been removed; use add!(::NamedTuple, ::Batch).
• ResultData(::Batch) and PostProcessor(::Batch) have been removed; use BatchData(::Batch).
• gemsplot's splitlabel is renamed splitgroup, and the combined = :bylabel option is renamed combined = :bygroup. The old names are no longer exported.
• customlogger!(::Batch, ::CustomLogger) has been removed; pass customlogger to process! instead.
• Several BatchProcessor accessors that required storing every ResultData (config_files, pathogens, settingdata, population_pyramid, setting_age_contacts, strategies, run_ids, runtime, allocations, extract, extract_unique, …) have been removed. Use simconfigs(batch) for configuration metadata, or process with keep_rundata = true and read per-run data via rundata(bp).

See the [batch tutorial](https://immidd.github.io/GEMS/stable/tut_batches/) for the updated workflow and migration examples.

---

Custom Individual Extensions

You can now attach arbitrary per-agent attributes to Individuals without modifying the core struct, by passing ind_extension to the Population or Simulation constructor. Extension fields behave exactly like built-in fields, and every existing GEMS function that accepts an ::Individual continues to work unchanged.

• Three ways to define extensions: (1) named columns — pass a vector of column names that already exist in the population DataFrame (e.g. ind_extension = [:my_attr]), whose values are read from each row and wrapped automatically; (2) a separate extension DataFrame joined by id (individuals absent from the table receive zero-filled values with a warning); (3) a factory function ind -> ext that computes each individual's extension from its base attributes — it may return any struct, including a @kwdef struct constructed by keyword arguments when the struct has many fields.
• Transparent field access: Custom attributes are read and written like core fields (ind.my_attr, ind.my_attr = 0.9). Access is implemented via getproperty/setproperty! overloads; for literal core-field names the dispatch is constant-folded and inlines to a single getfield, so built-in field access keeps its original performance. Extension data is held in a boxed extensions::Any slot, wrapped in an AutoExtension NamedTuple for the named-column/DataFrame paths.
• Population from a DataFrame: Simulation's population argument now also accepts a DataFrame directly (in addition to a path, identifier, or Population object), and Population(df; ind_extension = ...) builds individuals in parallel as before.
• Round-trips to DataFrame: dataframe(population) appends extension fields as additional columns after the base columns, taking the schema from the first extended individual.
• Collision safety: Extension field names must not clash with core Individual fields. Collisions are rejected at load time with an explicit error, preventing a custom field from silently shadowing a built-in attribute and keeping the population exportable.
• Type-stability caveat: Because the extensions slot is typed Any, reading custom fields is type-unstable. This is negligible in dynamically-dispatched code such as custom transmission_probability or sample_contacts! methods, but for heavy per-contact computation on custom fields you can recover full type stability with a function barrier.

See the [Custom Individual Extensions](https://immidd.github.io/GEMS/stable/tut_configfiles/#Custom-Individual-Extension) section of the config-file tutorial.

---

Intervention Engine Optimization

The intervention pipeline — the event queue, strategy callbacks, and per-tick setting iteration — has been reworked to be type-stable and allocation-free on the hot path. This adds a dependency on FunctionWrappers.jl.

• Tick-bucketed event queue: EventQueue has been rewritten from a sorted-vector priority queue (with O(N) searchsortedfirst + insert!) into a "calendar" queue of per-tick buckets. enqueue! is now an O(1) push! into the relevant tick bucket, and the queue is drained in tick order via a forward-advancing head pointer. New peektick and empty! (capacity-retaining) functions are provided; first is no longer part of the public queue API (use peek).
• Type-stable strategy callbacks: Strategy conditions, delays, and the per-measure process_measure calls are now wrapped in concrete FunctionWrapper types (IPredicate/SPredicate, IDelayFn/SDelayFn, IProcessFn/SProcessFn). MeasureEntry is now parameterized on the focal object type (Individual or Setting), keeping each strategy's measure vector concretely typed. The process_measure callback for each measure is built once at add_measure! time — where the concrete measure type is statically known — and stored on the event, turning the former dynamic process_measure dispatch into a fixed indirect call. IMeasureEvent/SMeasureEvent are now immutable structs carrying this prebuilt callback.
• Type-stable setting iteration: SettingsContainer now stores each setting type in a concretely-typed Vector{T} (the dict value type was widened from Vector{Setting} to Vector). A new foreach_setting_vector iterates the built-in setting types (BUILTIN_SETTING_TYPES) with full type stability and no dynamic dispatch, and a new settings(sim, ::Type{T}) accessor returns a typed Vector{T}. step!, open!, close!, and the tick triggers now route through these type-stable paths via function barriers.
• Compile-time debug switch: All @debug logging on the intervention hot path is now gated behind the INTERVENTION_DEBUG constant (default false), so the logging machinery is fully elided in normal runs. Flip it to true when debugging.
• FindMembers fast path: The selectionfilter is now wrapped as an IPredicate with a has_filter flag and a _select_all sentinel default. When no custom filter is supplied, the filtering pass is skipped entirely, and process_measure no longer rebuilds the member list multiple times.

Note for API users

condition(strategy) and condition(measure_entry) now return a wrapped predicate (IPredicate/SPredicate) rather than a bare Function. The wrappers are callable exactly like the original closures, but code that asserted condition(str) isa Function or compared the stored condition by identity will need updating.

---

Bugfixes

• Disease progression InexactError: Transition-time calculations across all progression categories (Asymptomatic, Symptomatic, Severe, Hospitalized, Critical) now use round(Int16, ...) instead of Int16(...). Previously, a sampled transition duration that wasn't already a whole number would throw an InexactError; the values are now rounded to the nearest tick. death_probability is also explicitly cast to Float64 before comparison, and DiseaseProgression's default keyword values are now Int16-typed.

---

Other Changes & API Cleanup

• Internal PostProcessor constructor removed: The undocumented 7-argument PostProcessor(simulation, population, infections, vaccinations, deaths, tests, quarantines) reconstruction constructor has been removed.
• Expanded test coverage: This release adds substantial new tests across interventions, settings, populations, simulations, batches, reporting, and post-processing — including the new tick-bucketed event queue, type-stable accessors, the Dict-based Simulation constructor, and the batch streaming pipeline.

---

This discussion was created from the release Release 1.1: Batch Rework, Custom Individuals & Intervention Overhaul.