Skip to content

perf(data-layer): add bounded batch reads and Blob client reuse [2/5] - #186

Open
Anthony Cintron (prbatero) wants to merge 1 commit into
prbatero/feat/perf-layer-loading-baselinefrom
prbatero/feat/perf-data-layer-core
Open

perf(data-layer): add bounded batch reads and Blob client reuse [2/5]#186
Anthony Cintron (prbatero) wants to merge 1 commit into
prbatero/feat/perf-layer-loading-baselinefrom
prbatero/feat/perf-data-layer-core

Conversation

@prbatero

Copy link
Copy Markdown
Contributor

Summary

Part 2 of 5 in the project layer-loading performance stack. Add reusable storage primitives that remove sequential amplification while enforcing one process-wide blocking-I/O budget.

Changes

  • Add native Cosmos/PostgreSQL keyed batch reads and bounded fallback reads.
  • Share one process-wide executor across metadata and artifact downloads.
  • Reuse Blob service clients and lazily cached user-delegation keys.
  • Match metadata types exactly so model cannot consume model_catalog.
  • Make artifact downloads path-safe and atomic.
  • Align read contracts across Blob, local filesystem, Cosmos DB, Data Lake, and PostgreSQL.

Testing

  • Full hastelib suite: 553 passed.
  • Black, isort, flake8, and detect-secrets passed.

Stack

Review Notes

Review the diff against the configured base branch. Logical data-layer calls are not Azure REST transaction counts; Blob GET volume remains proportional to records returned.

Add native and fallback keyed reads across metadata backends, exact metadata-type matching, one process-wide blocking I/O budget, reusable Blob clients and delegation keys, and atomic artifact downloads with regression coverage.
@prbatero
Anthony Cintron (prbatero) requested a review from a team as a code owner September 1, 2026 17:38
@prbatero Anthony Cintron (prbatero) changed the title perf(data-layer): add bounded batch reads and client reuse [2/5] perf(data-layer): add bounded batch reads and Blob client reuse [2/5] Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant