feat: search buildings by area/size, then download time series#33
Merged
Merged
Conversation
…download their time series Adds the ability to find a specific set of buildings by criteria beyond state/county/building_type -- e.g. "all office buildings under 10,000 sqft in the Denver metro area" -- and feed the result directly into process_building_time_series() to download time series only for those buildings. - county_name now accepts a list of county names (in addition to a single name or "All") in both ComStockProcessor and ResStockProcessor, so a metro area spanning multiple counties (e.g. Denver, Arapahoe, Jefferson, Adams, Douglas, Broomfield) can be queried in one call instead of requiring state="All" (over-fetching the whole state) or one call per county. - Add optional min_sqft/max_sqft constructor parameters to both processors, filtering on the in.sqft..ft2 column shared by ComStock and ResStock metadata. Implemented as a shared _apply_sqft_filter() helper on BuildingStockProcessor since the column name is identical across products. - Add scope_label()/sqft_label() helpers to building_stock_processor.py that build filesystem-safe, deterministic cache-filename components for the new county-list and sqft-range filters, so differently-scoped searches against the same state/building_type/upgrade don't collide on stale cached CSVs. Note: this changes the selected_metadata.csv cache filename format (adds a square-footage-range segment), so existing cached files from before this change will no longer be reused and will be regenerated. - process_metadata()'s docstring and the county-filtering logic in both processors now use isin() against the (possibly multi-value) county list instead of a single equality check. - Add tests/test_building_stock_processor.py for the new scope_label/ sqft_label/validate_release pure functions, plus integration tests in both existing test files for multi-county filtering, sqft filtering, cache filename uniqueness across filter combos, and the full "search metadata, then download time series for the matching buildings" workflow. - Document the new filters and workflow in README.md with a Denver-area example. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
nllong
added a commit
that referenced
this pull request
Jul 25, 2026
* chore: migrate from Poetry to uv, update dependencies, fix test markers
- Migrate pyproject.toml from Poetry to PEP 621 + uv (dependency-groups
for dev deps, tool.uv package = false since this is an app, not a
library). Drop the unneeded pathlib backport dependency (stdlib
covers this on Python 3.12+).
- Bump all runtime and dev dependencies to current stable majors
(pandas 3.x, requests 2.34, geopandas 1.1, pyarrow 25, pytest 9,
mypy 2.x, pre-commit 4.6, etc.) and regenerate uv.lock, replacing
poetry.lock.
- Fix a pandas 3.0 breaking change (GroupBy.apply no longer retains
grouping columns) that was causing the notebook step to fail with
KeyError: in.county_name on every dependency-bump CI run; switch
to GroupBy.sample() which is version-safe and avoids the issue.
- Fix mypy config: it pointed at a nonexistent your_module.py and
used python_version = 3.8, so it was silently never checking the
real code. Point it at comstock_processor.py/tests/, target 3.12,
and relax strict annotations for tests. This surfaced a real bug:
process_building_time_series() was annotated to return None but
actually returns a tuple of lists; fixed the annotation and added
missing type hints.
- Connect all tests to CI: 7 of 10 tests in
tests/test_comstock_processor.py had no unit/integration marker, so
--strict-markers plus -m unit / -m integration in CI silently never
ran them. Added the correct marker to every test so the full suite
executes.
- Update pre-commit config: bump ruff-pre-commit and nbstripout to
latest, add astral-sh/uv-pre-commit's uv-lock hook to keep uv.lock
in sync.
- Update CI workflow to install/run everything through uv
(astral-sh/setup-uv) instead of Poetry, which was also failing to
install on the Python 3.9 matrix job; drop the unsupported 3.9 job
(project requires Python >=3.12) in favor of 3.12/3.13.
- Update README installation/testing/commit instructions for uv.
Verified locally: uv run pytest -m unit, uv run pytest -m integration
(real network downloads), uv run mypy, uv run pre-commit run
--all-files, and a full notebook execution all pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: pin setup-uv to full version tag v9.0.0
Unpinned major-only v9 tag does not exist for astral-sh/setup-uv,
only full semver tags like v9.0.0.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* style: apply ruff-format to notebook, drop removed S320 ignore
The newer ruff-pre-commit (v0.15.22) reformats the notebook cell I
edited after the last local pre-commit run, and warns that the S320
rule referenced in ruff.toml ignore list no longer exists in this
ruff version. Fix both so pre-commit passes cleanly in CI.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: support the last three ComStock releases
ComStock is periodically republished with a new set of released data, and the
metadata file layout/paths have changed across releases (release_1 used a
single national metadata/baseline.parquet file; release_2 and release_3
partition metadata per state/county instead). This adds explicit,
configurable support for the last three releases so callers aren't locked to
a single hardcoded, increasingly stale release.
- Add a SUPPORTED_RELEASES registry (release_1/2/3, mapping to
comstock_amy2018_release_N) plus DEFAULT_RELEASE = release_3. All three
currently resolve to the 2025 OEDI mirror, which republishes every release
using the same metadata_and_annual_results/by_state_and_county partitioned
layout, so one code path now handles all three releases uniformly.
ComStockProcessor takes a new optional `release` kwarg (defaulting to the
latest), validated against the registry with a clear ValueError otherwise.
- Rewrite process_metadata() to discover the relevant state/county partitions
via the public S3 list-objects-v2 API (stdlib xml.etree, no new
dependency), download them in parallel with the same
ThreadPoolExecutor/caching pattern used for time series files, concatenate,
and apply the same county/building-type filters as before. Partition
downloads and the final selected_metadata CSV are namespaced by release so
switching releases doesn't collide with or reuse another release's cache.
- process_building_time_series() needed no structural changes: the
timeseries_individual_buildings/by_state layout is identical across all
three releases.
- Fix 01_data_sampling_example.ipynb, which broke against the new default
release: the by_state_and_county metadata schema renamed several columns
with explicit unit suffixes (e.g. in.sqft -> in.sqft..ft2,
out.electricity.total.energy_consumption ->
...energy_consumption..kwh, electricity_bill_median..usd split into
_median_high/_median_low, electricity_bill_number_of_rates ->
electricity_bill_num_bills). Time series column names are unchanged.
- Update tests: add unit tests for release validation/URL construction,
parametrize metadata download/filter tests across all three releases, and
gate the state="All" test behind TEST_DATA=true since "All" now means
downloading every state's and county's partition files instead of a single
cached national parquet.
- Document the release parameter and supported releases in README.md.
Note: this changes the previous hardcoded default from
comstock_amy2018_release_1 (2024 flat layout) to release_3 (2025
partitioned layout) — existing callers that didn't pass `release` will now
get the newest release's data/schema.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* perf: parallelize metadata partition discovery/downloads, bound CI runtime
Use a higher I/O-bound worker count (IO_WORKERS=16) for S3 listing and
metadata partition downloads instead of the CPU-bound worker count used for
time series downloads; on CI runners with few cores this previously limited
metadata downloads to essentially one file at a time. Also parallelize the
per-state county listing (previously sequential) for state="All". Bound the
CI TEST_DATA integration step with timeout-minutes + continue-on-error,
since state="All" now downloads a very large number of small files and
should not be allowed to hang the job indefinitely.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test: mock state discovery for the state='All' test instead of a real full download
The previous TEST_DATA-gated test_all_state_filter genuinely downloaded every
state's/county's metadata partition (thousands of files) when TEST_DATA=true,
which took long enough in CI to hit the platform's workflow run limits.
Mock ComStockProcessor._available_states down to two small states (DE, RI)
so the test still exercises the real state='All' discover/download/
concatenate/filter code path, but stays fast and deterministic. This removes
the need for the TEST_DATA skip and keeps the test in the regular
integration suite.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: support comparing buildings across measure packages
Each ComStock "upgrade" is a different energy-efficiency measure package applied
to the same baseline building sample. Every release also publishes an
upgrades_lookup.json (upgrade id -> package name) and a measure_name_crosswalk.csv
(a stable measure_id crosswalked to the upgrade id/name used for that measure in
that release and earlier releases, since upgrade ids/measures are release-specific
and not stable across releases).
- Add ComStockProcessor.list_upgrades(save_dir) to download/cache
upgrades_lookup.json and return {upgrade_id: package_name} for the configured
release.
- Add ComStockProcessor.get_measure_crosswalk(save_dir) to download/cache
measure_name_crosswalk.csv as a DataFrame, and find_upgrade_id(save_dir,
measure_id, target_release=None) to look up the upgrade id for a stable
measure_id in a specific release, raising a clear error if that release isn't
covered by the currently loaded crosswalk (a release's crosswalk only covers
itself and earlier releases -- release_3's crosswalk covers all three
currently-supported releases).
- Refactor process_metadata() into a thin wrapper around a new
_download_metadata_for_upgrade(save_dir, upgrade) helper parameterized by
upgrade instead of hardcoded self.upgrade, and add
process_metadata_for_upgrades(save_dir, upgrades=None) which downloads and
combines metadata for multiple upgrades (defaulting to every upgrade from
list_upgrades()) into one DataFrame. Every partition already includes an
`upgrade` id and `in.upgrade_name` column, so grouping the combined result by
bldg_id lets you compare a building's results across packages.
- Add unit tests (release validation, mocked default-to-every-upgrade behavior)
and integration tests (list_upgrades, get_measure_crosswalk, find_upgrade_id
across releases, and process_metadata_for_upgrades with an explicit small
upgrade list for Delaware/SmallOffice).
- Document the new methods and a building-comparison usage example in
README.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: support ResStock and handling of multifamily buildings
Adds support for NREL's residential building stock dataset, ResStock, alongside
the existing ComStock support, and documents/handles ResStock's per-dwelling-unit
representation of multifamily buildings.
Background: ResStock is hosted on the same OEDI data lake as ComStock
(nrel-pds-building-stock/end-use-load-profiles-for-us-building-stock/{year}/resstock_*),
but differs in several ways: metadata is partitioned only by state (not
state+county), building types are residential housing categories (not commercial),
county names don't include a state prefix, and the measure crosswalk is an Excel
file, not csv. Multifamily buildings are represented at the *dwelling unit* level:
one metadata row is one simulated unit (not a whole building), with
weight/in.units_represented for population scaling and
in.geometry_building_number_units_mf/in.geometry_building_horizontal_location_mf/
in.geometry_building_level_mf describing the unit's context within its building.
- Extract a shared BuildingStockProcessor base class (building_stock_processor.py)
containing the S3 listing/downloading, upgrade package lookup
(list_upgrades), and time series download logic that is identical between
ComStock and ResStock. ComStockProcessor now subclasses it, keeping its own
state+county partitioned metadata logic and csv-based measure crosswalk.
BuildingStockRelease replaces the former ComStock-only ComStockRelease
dataclass (same shape, shared name); public symbols used elsewhere
(SUPPORTED_RELEASES, DEFAULT_RELEASE, ComStockProcessor) are unaffected.
- Add ResStockProcessor (resstock_processor.py), a new subclass with its own
SUPPORTED_RELEASES (release_1 -> resstock_amy2018_release_1; ResStock hasn't
been remastered across multiple releases into one consistent layout yet, unlike
ComStock, so only this one is supported for now), state-only partitioned
metadata download, RESSTOCK_BUILDING_TYPES validation (residential housing
categories including the two multifamily categories), county_name filtering
without a state prefix, and an xlsx-based get_measure_crosswalk/find_upgrade_id.
process_metadata_for_upgrades works the same as ComStockProcessor's, for
comparing a dwelling unit across measure packages.
- Add openpyxl as a dependency (needed to read ResStock's .xlsx measure
crosswalk).
- Add tests/test_resstock_processor.py mirroring the ComStock test suite:
unit tests for release/building-type validation and mocked
default-to-every-upgrade behavior, and integration tests for metadata
download/filtering/caching (including both multifamily categories and the
county_name-without-state-prefix format), list_upgrades, get_measure_crosswalk,
find_upgrade_id, process_metadata_for_upgrades, and time series downloading.
All existing ComStockProcessor tests still pass unchanged after the refactor.
- Document ResStockProcessor, its differences from ComStockProcessor, and how to
handle multifamily buildings in README.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* style: apply ruff-format line-length collapsing to new tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: support comparing buildings across measure packages (#31)
Each ComStock "upgrade" is a different energy-efficiency measure package applied
to the same baseline building sample. Every release also publishes an
upgrades_lookup.json (upgrade id -> package name) and a measure_name_crosswalk.csv
(a stable measure_id crosswalked to the upgrade id/name used for that measure in
that release and earlier releases, since upgrade ids/measures are release-specific
and not stable across releases).
- Add ComStockProcessor.list_upgrades(save_dir) to download/cache
upgrades_lookup.json and return {upgrade_id: package_name} for the configured
release.
- Add ComStockProcessor.get_measure_crosswalk(save_dir) to download/cache
measure_name_crosswalk.csv as a DataFrame, and find_upgrade_id(save_dir,
measure_id, target_release=None) to look up the upgrade id for a stable
measure_id in a specific release, raising a clear error if that release isn't
covered by the currently loaded crosswalk (a release's crosswalk only covers
itself and earlier releases -- release_3's crosswalk covers all three
currently-supported releases).
- Refactor process_metadata() into a thin wrapper around a new
_download_metadata_for_upgrade(save_dir, upgrade) helper parameterized by
upgrade instead of hardcoded self.upgrade, and add
process_metadata_for_upgrades(save_dir, upgrades=None) which downloads and
combines metadata for multiple upgrades (defaulting to every upgrade from
list_upgrades()) into one DataFrame. Every partition already includes an
`upgrade` id and `in.upgrade_name` column, so grouping the combined result by
bldg_id lets you compare a building's results across packages.
- Add unit tests (release validation, mocked default-to-every-upgrade behavior)
and integration tests (list_upgrades, get_measure_crosswalk, find_upgrade_id
across releases, and process_metadata_for_upgrades with an explicit small
upgrade list for Delaware/SmallOffice).
- Document the new methods and a building-comparison usage example in
README.md.
Co-authored-by: Nicholas Long <nicholas.long@nrel.gov>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: search buildings by multi-county area and square footage, then download their time series (#33)
Adds the ability to find a specific set of buildings by criteria beyond
state/county/building_type -- e.g. "all office buildings under 10,000 sqft in
the Denver metro area" -- and feed the result directly into
process_building_time_series() to download time series only for those
buildings.
- county_name now accepts a list of county names (in addition to a single
name or "All") in both ComStockProcessor and ResStockProcessor, so a metro
area spanning multiple counties (e.g. Denver, Arapahoe, Jefferson, Adams,
Douglas, Broomfield) can be queried in one call instead of requiring
state="All" (over-fetching the whole state) or one call per county.
- Add optional min_sqft/max_sqft constructor parameters to both processors,
filtering on the in.sqft..ft2 column shared by ComStock and ResStock
metadata. Implemented as a shared _apply_sqft_filter() helper on
BuildingStockProcessor since the column name is identical across products.
- Add scope_label()/sqft_label() helpers to building_stock_processor.py that
build filesystem-safe, deterministic cache-filename components for the new
county-list and sqft-range filters, so differently-scoped searches against
the same state/building_type/upgrade don't collide on stale cached CSVs.
Note: this changes the selected_metadata.csv cache filename format (adds a
square-footage-range segment), so existing cached files from before this
change will no longer be reused and will be regenerated.
- process_metadata()'s docstring and the county-filtering logic in both
processors now use isin() against the (possibly multi-value) county list
instead of a single equality check.
- Add tests/test_building_stock_processor.py for the new scope_label/
sqft_label/validate_release pure functions, plus integration tests in both
existing test files for multi-county filtering, sqft filtering, cache
filename uniqueness across filter combos, and the full "search metadata,
then download time series for the matching buildings" workflow.
- Document the new filters and workflow in README.md with a Denver-area
example.
Co-authored-by: Nicholas Long <nicholas.long@nrel.gov>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Nicholas Long <nicholas.long@nrel.gov>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the ability to search for a set of buildings by criteria beyond
state/county_name/building_type— specifically a multi-county "area" query and a square footage range — and feed the result directly intoprocess_building_time_series()to download time series data only for the matching buildings.Motivating example
Denver's metro area spans multiple counties (Denver, Arapahoe, Jefferson, Adams, Douglas, Broomfield), so a single
county_namestring couldn't express that query before — you'd either over-fetch the entire state (county_name="All") or callprocess_metadata()once per county and concatenate. There was also no way to filter by building size.What changed
county_namenow accepts a list of county names (in addition to a single string or"All") in bothComStockProcessorandResStockProcessor. Filtering usesisin()against the (possibly multi-value) list.min_sqft/max_sqftconstructor parameters to both processors, filtering on thein.sqft..ft2column (identical name in both products). Implemented as a shared_apply_sqft_filter()helper onBuildingStockProcessor.scope_label()/sqft_label()helpers (building_stock_processor.py) that build deterministic, filesystem-safe cache-filename components for the new filters, so different searches against the same state/building_type/upgrade don't collide on stale cached CSVs...._selected_metadata.csvcache filename format (adds a square-footage-range segment) — existing cached files from before this change won't be reused and will regenerate.tests/test_building_stock_processor.pyfor the purescope_label/sqft_label/validate_releasefunctions, plus integration tests in both existing test files covering multi-county filtering, sqft filtering, cache-filename uniqueness across filter combinations, and the full search → download-time-series workflow.Validation
uv run pytest tests/ -m unit✅ (29 passed)uv run pytest tests/ -m integration✅ (33 passed, ~1–3 min — full existing ComStock/ResStock suites unchanged + new multi-county/sqft/search-then-download tests for Delaware)uv run mypy✅ (7 source files)uv run pre-commit run --all-files✅