feat: support ResStock and multifamily building handling#32
Open
nllong wants to merge 2 commits into
Open
Conversation
Adds support for NREL's residential building stock dataset, ResStock, alongside
the existing ComStock support, and documents/handles ResStock's per-dwelling-unit
representation of multifamily buildings.
Background: ResStock is hosted on the same OEDI data lake as ComStock
(nrel-pds-building-stock/end-use-load-profiles-for-us-building-stock/{year}/resstock_*),
but differs in several ways: metadata is partitioned only by state (not
state+county), building types are residential housing categories (not commercial),
county names don't include a state prefix, and the measure crosswalk is an Excel
file, not csv. Multifamily buildings are represented at the *dwelling unit* level:
one metadata row is one simulated unit (not a whole building), with
weight/in.units_represented for population scaling and
in.geometry_building_number_units_mf/in.geometry_building_horizontal_location_mf/
in.geometry_building_level_mf describing the unit's context within its building.
- Extract a shared BuildingStockProcessor base class (building_stock_processor.py)
containing the S3 listing/downloading, upgrade package lookup
(list_upgrades), and time series download logic that is identical between
ComStock and ResStock. ComStockProcessor now subclasses it, keeping its own
state+county partitioned metadata logic and csv-based measure crosswalk.
BuildingStockRelease replaces the former ComStock-only ComStockRelease
dataclass (same shape, shared name); public symbols used elsewhere
(SUPPORTED_RELEASES, DEFAULT_RELEASE, ComStockProcessor) are unaffected.
- Add ResStockProcessor (resstock_processor.py), a new subclass with its own
SUPPORTED_RELEASES (release_1 -> resstock_amy2018_release_1; ResStock hasn't
been remastered across multiple releases into one consistent layout yet, unlike
ComStock, so only this one is supported for now), state-only partitioned
metadata download, RESSTOCK_BUILDING_TYPES validation (residential housing
categories including the two multifamily categories), county_name filtering
without a state prefix, and an xlsx-based get_measure_crosswalk/find_upgrade_id.
process_metadata_for_upgrades works the same as ComStockProcessor's, for
comparing a dwelling unit across measure packages.
- Add openpyxl as a dependency (needed to read ResStock's .xlsx measure
crosswalk).
- Add tests/test_resstock_processor.py mirroring the ComStock test suite:
unit tests for release/building-type validation and mocked
default-to-every-upgrade behavior, and integration tests for metadata
download/filtering/caching (including both multifamily categories and the
county_name-without-state-prefix format), list_upgrades, get_measure_crosswalk,
find_upgrade_id, process_metadata_for_upgrades, and time series downloading.
All existing ComStockProcessor tests still pass unchanged after the refactor.
- Document ResStockProcessor, its differences from ComStockProcessor, and how to
handle multifamily buildings in README.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds support for NREL's residential building stock dataset, ResStock, alongside the existing ComStock support, and handles ResStock's representation of multifamily buildings.
Background: where ResStock lives, and multifamily handling
ResStock is hosted on the same OEDI data lake as ComStock (
nrel-pds-building-stock/end-use-load-profiles-for-us-building-stock/{year}/resstock_*), but differs in a few important ways I confirmed by inspecting the bucket directly:state=DE/DE_upgrade0.parquet), not state+county like ComStock.Single-Family Detached,Multi-Family with 5+ Units, etc.), stored inin.geometry_building_type_recs.in.county_namehas no state prefix ("Kent County", not"DE, Kent County").measure_name_crosswalk_res_{year}_{release}.xlsx), not csv.resstock_amy2018_release_1has been remastered into the current, consistent 2025 layout so far — the newerresstock_amy2018_release_2still lives at its original (differently-structured) 2024 path, so it isn't supported yet.Multifamily buildings: ResStock simulates individual dwelling units, not whole buildings — a "Multi-Family with 5+ Units" metadata row is one apartment unit, not the building. There's no shared "building id" tying units back to a real building; each is an independently sampled, weighted record. Relevant columns:
in.geometry_building_number_units_mf(units in that unit's building),in.geometry_building_horizontal_location_mf/in.geometry_building_level_mf(the unit's position, affecting heat transfer with neighbors), andweight/in.units_represented(population scaling).What changed
BuildingStockProcessor(building_stock_processor.py): a shared base class with the S3 listing/downloading,list_upgrades(), andprocess_building_time_series()logic that's identical between ComStock and ResStock (same bucket, same timeseries layout, sameupgrades_lookup.jsonconvention).ComStockProcessornow subclasses it, unchanged behaviorally — all existing tests pass without modification.ResStockProcessor(resstock_processor.py):SUPPORTED_RELEASES = {"release_1": ...}(only the one confirmed-working layout, for now).RESSTOCK_BUILDING_TYPES— the 5 validin.geometry_building_type_recsvalues, validated in the constructor.process_metadata()/process_metadata_for_upgrades()(simpler than ComStock's, no county sub-listing needed).county_namefiltering without the state prefix,get_measure_crosswalk()/find_upgrade_id()adapted to the xlsx format.openpyxldependency (to read the xlsx crosswalk).tests/test_resstock_processor.pymirrors the ComStock test suite — release/building-type validation, mocked default-upgrades behavior, and integration tests for metadata download/filtering (including both multifamily categories and county-name-without-prefix), caching,list_upgrades,get_measure_crosswalk,find_upgrade_id,process_metadata_for_upgrades, and time series downloads.Validation
uv run pytest tests/ -m unit✅ (16 passed — release/building-type validation, mocked behaviors, for both ComStock and ResStock)uv run pytest tests/ -m integration✅ (27 passed, ~1 min — full ComStock suite unchanged + new ResStock metadata/multifamily/crosswalk/timeseries tests for Delaware)uv run mypy✅ (6 source files)uv run pre-commit run --all-files✅find_upgrade_id, and time series downloads all work end-to-end against the live OEDI bucket