Repository navigation
Releases: awindmann/SensorFault-Bench
Releases · awindmann/SensorFault-Bench
Release list
SensorFault-Bench 1.0.1
Patch release that makes dataset parsing and scoring independent of the
platform. All results in the paper and the paper artifact remain valid, since
the datasets and every reported value are unchanged.
Fixed
- Validation with
--require-checksumsaccepts the downloadedETTh1.csvon
every platform. Every dataset CSV file is now parsed with correct rounding,
whereas the default pandas parser rounds long decimals differently from
platform to platform. This changes the recorded content checksum of ETTh1
but not the values that models train on. scripts/preprocess_penmanshiel_hourly_wt08.pyreproduces the published
penmanshiel_hourly_wt08.parqueton every platform, so a rebuild from the
raw SCADA files passes--require-checksums. Previously, the rebuilt values
depended on the platform.- Analysis and the fixed-channel-fraction evaluation read the logged per-sample
errors exactly as written. Before, the last digit of full-precision outputs
could differ between platforms, while reported values were not affected. - A Git checkout on Windows keeps the committed bytes of
paper_artifact/and
configs/baseline_hparams.yaml, so these files match the checksums that the
manifests inpaper_artifact/record.
SensorFault-Bench 1.0.0
Camera-ready release for the NeurIPS 2026 Evaluations & Datasets Track paper
"Benchmarking Sensor-Fault Robustness in Forecasting". The results of the
default configuration are unchanged, as are its scored scenarios, selection
rules, and scoring.
Added
paper_artifact/supplementary_tables/, the tables that the paper added in
its camera-ready revision. ItsREADME.mddescribes each table.data/croissant/, the Croissant metadata of the two curated datasets with
their Responsible AI fields.- The sensitivity scenarios
negative_drift,negative_spike, and
local_timing_warp. - The severity sampling distribution
beta_1_2, which
PERTURBATION_SEVERITY_DISTRIBUTIONselects. The default staysuniform. docs/benchmark_protocol.md,docs/sensitivity_analyses.md, a results
schema and troubleshooting indocs/reproducing_benchmark.md, and paired
transfer inputs indocs/extending_methods.md.CITATION.cff, this changelog, and continuous integration.docs/using_fault_operators.md, a guide to uses of the fault operators
outside the benchmark, such as pretraining or stress tests calibrated to
observed faults. The protocol section "Fault Exposure in Training" asks
reports to name the fault operators that a model saw in training, including
pretraining.notebooks/tutorial/quickstart.ipynb, a CPU tutorial that shows the scored
scenarios, runs a small benchmark, and reads the reported quantities.configs/cli_help.yaml, the help text of every command-line option. The
--helpoutput of the three entry points groups the options by topic and
shows their defaults.
Changed
- The README is restructured around the key findings, the results of the
default configuration, and a guide to evaluating a new model or method.
data/README.mdadds download commands for the two curated datasets. - The two curated datasets are published as Zenodo records with Croissant 1.1
metadata, from whichdata/README.mddownloads them. Their Parquet and CSV
files are byte-identical to version 2 of the Kaggle records of the review
period. - The paper artifact is version 2,
sensorfault-bench-neurips2026-paper-artifact-v2. - The folders in
paper_artifact/tables/carry the name of their evaluated
setting, for exampledefault_configuration/instead of
primary_clean_2026-04-26_10k/.paper_artifact/MANIFEST.jsonrecords the
export date and source path of each folder. The tables are unchanged. - The figure scripts and PDFs in
paper_artifact/R/match the camera-ready
figures. Their inputs are unchanged. paper_artifact/README.mdmaps each section of the paper to its bundled
results and the commands that reproduce them, and states that the paper
corresponds to releasev1.0.0.- The compute accounting moved from
evidence/compute_accounting/to
paper_artifact/compute_accounting/. It also covers the seed reruns, ETTh1
task settings, and sensitivity checks of the camera-ready revision. Its values
for the default configuration are unchanged.
scripts/compute_accounting_ledger.pywrites it from the campaigns listed in
configs/reporting/compute_accounting.yaml. - The extension guides describe the interfaces, work through one example each,
and list every configuration entry that a new architecture, method, or
dataset needs.docs/extending_models.mdshows how to evaluate a model whose
weights were trained elsewhere. - The architecture families of the analysis plots are configured in
configs/benchmark_scope.yaml, and the trajectory legends use the method
labels ofconfigs/reporting/core_figures.yaml. Analysis therefore accepts
new architectures and methods without code changes. instantiate_perturbationsindata/data_module.pyis now public.TSDataModuleno longer acceptsseverity_laws.
PERTURBATION_SEVERITY_DISTRIBUTIONsets the severity sampling distribution.DatasetSpecrequiresinput_channels.
Fixed
- Forecast replay binds the same randomized smoothing noise to each window as
degradation scoring does.scripts/render_forecast_plots.pyand the
rendering of extreme forecasts inrun_analysis.pytherefore reproduce the
scored forecasts of randomized smoothing runs instead of failing the score
check. - With
STRICT_IID: true, every sampled window lies inside its own block
together with its target horizon. No two windows therefore share a time step.
The results of the default configuration are unchanged because the option is
off by default and degradation evaluation rejects it. scripts/build_paper_artifact.py --forcebuilds the new bundle in a staging
directory and replaces the existing bundle only after the build succeeds. A
failed build leaves the previous bundle unchanged.- The artifact builder and the test suite run on Python 3.10, the minimum
version declared inpyproject.toml. Continuous integration tests Python
3.10, 3.11, and 3.12. - Training, testing, analysis, and forecast replay stop when the input or
target channel order recorded on a run differs from the dataset
specification. The data configuration signature compares channel sets, so a
run trained under another channel order was previously scored with the wrong
channel mapping. Training also counted such a run as a completed candidate. - Analysis and repeated testing accept runs that were reevaluated with fewer
scenarios, asdocs/sensitivity_analyses.mddescribes. They previously
stopped on the metrics that the earlier evaluation had logged for the dropped
scenarios. run_training.pytrainsbaselinefirst and wrapping methods last, whatever
the order of--method, so a method listed beforebaselineno longer fails
on a new experiment.run_analysis.pystops before reading any runs and
names the missingbaselinewhen--methodomits it.- Training stops with a clear error when the validation split holds fewer
windows than one batch. Validation scores full batches only, so such a split
previously ended in an error about the missing early-stopping metric. - Testing keeps the end time of each evaluated run. Because the end time breaks
ties between equal validation scores, the new end time that evaluation used
to record could make a repeated test select another winner after
--refresh-selection-metrics. - Scoring accepts bootstrap percentile intervals that exclude the point
estimate, which a smallBOOTSTRAP_CI_RESAMPLEScan produce. Such intervals
previously stopped scoring. - Fault augmentation runs with data loader workers under the
spawnstart
method, the default on macOS and Windows. Its severity sampler previously
could not be passed to the workers. - Randomized training stops when its noise level is not finite, for example
nan. Such a run previously trained without noise.