Support reading skims from Parquet files, in addition to OMX

[Copilot]

Skims can currently only be read from OMX files. Parquet storage is significantly more compact (~30% smaller) and can be read faster, so this adds Parquet as an alternative input format, auto-detected per-file, with full backward compatibility for OMX.

New Parquet skim reader (activitysim/core/skim_parquet.py)

• ParquetSkimFile inspects a parquet skim file's first two columns (origin, destination) to derive the zone id set and matrix shape; remaining columns are treated as named skim matrices, using the same naming/time-period conventions as OMX (e.g. DIST, DRV_COM_WLK_BOARDS__AM).
• Detects whether rows are dense (every OD pair present) and sorted row-major or column-major, using a fast reshape path in that case.
• Falls back to a sparse read path (scatter into a dense matrix) for incomplete or unsorted-dense-but-actually-sparse data; sparse data may be sorted or unsorted.
• Raises ValueError if data is dense (complete OD coverage) but not sorted row-major/column-major, since it can't be read efficiently or unambiguously in that state.

Integration (activitysim/core/skim_dict_factory.py)

• SkimInfo.load_skim_info and _read_skims_from_omx now branch per-file on extension (.parquet/.pq vs OMX), so OMX and Parquet files can be freely mixed under the same skim tag.
• Each file's zone ids/layout are validated independently — files are not assumed to share row ordering — while shape and zone id sets are still checked for consistency across all files for a given skim tag.
• No changes needed in los.py: users just reference .parquet filenames directly in network_los.yaml, e.g.:

taz_skims:
  - z1_taz_skims_part1.parquet
  - z1_taz_skims_part2.parquet

Scope

• Applies to the legacy NumpyArraySkimFactory/MemMapSkimFactory path (SkimDict).
• The sharrow-based SkimDataset path still relies on the external sharrow package's sh.dataset.from_omx_3d and does not yet support Parquet input; extending that would require changes upstream in sharrow.

Docs

• TAZ_Settings.omx and SkimInfo docstrings updated to describe Parquet input expectations (dense/sparse layout, column conventions).

[jpn--]

col_major_orig is the same as row_major_dest, there is no need to create this array twice.

[Copilot]

Fixed in 288c238 — col-major orig/dest now reuse row_major_dest/row_major_orig instead of recreating equivalent arrays.

[jpn--]

col_major_dest is the same as row_major_orig, there is no need to create this array twice.

[Copilot]

Fixed in 288c238 — same fix, reusing row_major_orig/row_major_dest for the col-major check.

[Copilot]

Pull request overview

Adds Parquet as an additional skim storage/input format for the legacy SkimDict pipeline, auto-detected per file by extension, and extends tests/config fixtures to validate Parquet-based LOS loading.

Changes:

• Introduce activitysim/core/skim_parquet.py with ParquetSkimFile for inspecting and reading Parquet skim tables (dense row/col-major and sparse).
• Extend SkimInfo.load_skim_info / _read_skims_from_omx to mix OMX and Parquet skim files under a single skim tag.
• Add unit/integration tests and canonical test configs validating Parquet skim loading (single file and multi-file scenarios).

Reviewed changes

Copilot reviewed 9 out of 12 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
activitysim/core/skim_parquet.py	New Parquet skim reader/inspector used by legacy skim loading.
activitysim/core/skim_dict_factory.py	Adds Parquet branching + caching during skim discovery and loading.
activitysim/core/configuration/network.py	Documents Parquet skim input expectations under TAZ_Settings.
activitysim/core/test/test_skim_parquet.py	New unit tests for dense/sparse Parquet layout detection and reading.
activitysim/core/test/test_los.py	New LOS integration tests exercising Parquet skim inputs.
activitysim/core/test/los/configs_1z_parquet/settings.yaml	New canonical config for 1-zone Parquet skim test.
activitysim/core/test/los/configs_1z_parquet/network_los.yaml	New network LOS config pointing to .parquet skims.
activitysim/core/test/los/configs_1z_parquet_multi/settings.yaml	New canonical config for multi-file Parquet skim test.
activitysim/core/test/los/configs_1z_parquet_multi/network_los.yaml	New LOS config pointing to two Parquet skim parts.

Comments suppressed due to low confidence (1)

activitysim/core/skim_parquet.py:138

• Same as above: pyarrow.Table.column(...) is not reliably name-addressable across the supported pyarrow range. Indexing the table by column name is the stable API.

        values = table.column(column_name).to_numpy(zero_copy_only=False)
        if dtype is not None:

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.