Skip to content

Rebuild NILMbench as a reproducible real-data benchmark - #1

Merged
nipunbatra merged 3 commits into
mainfrom
codex/rebuild-benchmark
Jul 17, 2026
Merged

Rebuild NILMbench as a reproducible real-data benchmark#1
nipunbatra merged 3 commits into
mainfrom
codex/rebuild-benchmark

Conversation

@nipunbatra

Copy link
Copy Markdown
Member

Summary

  • replace order-dependent benchmark notebooks with an installable nilmbench CLI
  • encode typed historical and corrected T1/T2/T3 task profiles in TOML
  • use NILM Metadata as the source of truth for appliance taxonomy and synonyms, while recording exact resolved appliance, meter/MeterGroup, AC power type, coverage, and shared-circuit contamination
  • separate legacy fixed-10-W F1 from the paper's appliance-specific thresholds
  • add deterministic seeds, resumable and identity-safe Optuna studies, structured JSON/CSV results, and source/runtime/container provenance
  • add separate CPU-smoke and pinned PyTorch 2.6 / CUDA 12.4 benchmark images
  • register PatchTST and pin its exact model commit from Add PatchTST sequence-to-point disaggregator nilmtk-contrib#100
  • document discrepancies recovered from the executed Ramanujan notebooks

Protocol findings

The recovered notebooks and paper disagree in several places: REDD T1 asks for data before the canonical file begins; REFIT T1 requests an unavailable microwave; REDD T2 MAE uses building 4 while the F1 notebook and paper use building 6; and notebook F1 uses NILMTK's fixed 10 W threshold although the paper specifies appliance thresholds. Historical profiles preserve those inputs for forensic reproduction. Corrected profiles use the paper splits, strict coverage, per-appliance alignment, and explicit thresholds.

NILM Metadata remains responsible for semantic interoperability. NILMbench adds only experiment-level configuration and provenance; it does not maintain a competing appliance alias table.

Verification

  • pytest: 9 passed
  • ruff check src tests: passed
  • uv lock --check: passed
  • docker compose config --quiet: passed
  • wheel and sdist: built and inspected
  • all eight corrected T1/T2/T3 tasks: passed read-only preflight on checksummed REDD, REFIT, and UK-DALE HDF5 data on Ramanujan
  • real REDD PatchTST CUDA smoke on A100-SXM4-80GB: 1 epoch, 1,024 train/test samples, 20.98 s, MAE 64.27 W, F1 0.6675 at the 50 W fridge threshold

Dependency

This PR is intentionally stacked on nilmtk/nilmtk-contrib#100 and pins commit f152f9f1783161bced09da7239d7ad57f4196a9b in the Python lock and container workflow.

@nipunbatra
nipunbatra merged commit d7d1b55 into main Jul 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant