Rr 61 polarslazyframe by ryanraaschCDC · Pull Request #65 · CDCgov/cfa-dataops

ryanraaschCDC · 2026-03-19T16:24:20Z

No description provided.

Copilot

Pull request overview

Adds Polars LazyFrame support to BlobEndpoint.get_dataframe() to enable lazy loading (via scan_*) for certain Azure-hosted datasets.

Changes:

Introduces a pl_lazy fast-path that uses pl.scan_parquet / pl.scan_csv against az:// URIs.
Adds Azure managed-identity credential wiring for Polars cloud reads.
Removes eager-to-lazy conversion for CSV in the eager path (now handled by the scan_csv early return).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR extends the data catalog’s BlobEndpoint.get_dataframe() to support returning a Polars LazyFrame for certain file types by using Polars’ lazy scanning against Azure Blob Storage.

Changes:

Expands get_dataframe() return type to include pl.LazyFrame.
Adds a pl_lazy fast-path that uses pl.scan_parquet / pl.scan_csv with Azure credential plumbing.
Removes the previous df.lazy() conversion for the eager CSV Polars path (since CSV lazy now uses scanning).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

damonbayer · 2026-03-19T18:19:12Z

+                )
+                self.ledger_entry(action="read")
+                return df
+            elif file_ext == "csv":


I think there is a strong opportunity to generalize all of the polars reading code into something with a variable reading function. Ie you can pass pl.read_csv or pl.scan_csv as an argument to more general function.

Copilot

Pull request overview

This PR updates BlobEndpoint.get_dataframe() to support returning a Polars LazyFrame backed by Azure Blob Storage scanning, while also tightening typing via Literal overloads and expanding supported output aliases.

Changes:

Add output="pl_lazy" / "lazy" mode that uses pl.scan_* against az:// URIs for parquet/csv/ndjson.
Replace the old pl_lazy: bool flag with typed output variants and overloads for better return-type inference.
Extend JSON-lines handling to treat ndjson and jsonl equivalently.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

ryanraaschCDC · 2026-03-27T15:33:34Z

@damonbayer please re-review. I made all your updates except for generalizing all the polars reading code. Let me know if that's something you need implemented and I can get that added.

damonbayer

The type hints are not working correctly, but it's not due to anything in this PR. Will open a separate issue. Thanks @ryanraaschCDC!

…61-polarslazyframe

ryanraaschCDC added 2 commits March 18, 2026 21:53

change lazyframe usage

8fca68b

add returns to pl lazy

847aa83

ryanraaschCDC requested a review from Copilot March 19, 2026 16:24

ryanraaschCDC linked an issue Mar 19, 2026 that may be closed by this pull request

Support loading datasets as polars.LazyFrame #61

Closed

Copilot AI reviewed Mar 19, 2026

View reviewed changes

ryanraaschCDC and others added 3 commits March 19, 2026 09:30

Potential fix for pull request finding

eba4f4a

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

42aab78

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

5757744

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

ryanraaschCDC requested a review from Copilot March 19, 2026 16:33

Copilot AI reviewed Mar 19, 2026

View reviewed changes

Comment thread cfa/dataops/catalog.py

Comment thread cfa/dataops/catalog.py Outdated

Comment thread cfa/dataops/catalog.py Outdated

Comment thread cfa/dataops/catalog.py Outdated

Comment thread cfa/dataops/catalog.py Outdated

Comment thread cfa/dataops/catalog.py

ryanraaschCDC and others added 4 commits March 19, 2026 09:37

Potential fix for pull request finding

50c188c

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

6bb69a6

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

f4dcb2c

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

add json capability

8162dee

ryanraaschCDC requested a review from damonbayer March 19, 2026 17:15

damonbayer reviewed Mar 19, 2026

View reviewed changes

Comment thread cfa/dataops/catalog.py Outdated

damonbayer reviewed Mar 19, 2026

View reviewed changes

Comment thread cfa/dataops/catalog.py

damonbayer reviewed Mar 19, 2026

View reviewed changes

Comment thread cfa/dataops/catalog.py Outdated

damonbayer reviewed Mar 19, 2026

View reviewed changes

ryanraaschCDC added 2 commits March 20, 2026 20:04

update pl_lazy to output

fbb554e

add jsonl and ndjson for lazy

e0b6970

ryanraaschCDC requested a review from Copilot March 23, 2026 17:25

Copilot started reviewing on behalf of ryanraaschCDC March 23, 2026 17:25 View session

Copilot AI reviewed Mar 23, 2026

View reviewed changes

Comment thread cfa/dataops/catalog.py

Comment thread cfa/dataops/catalog.py

Comment thread cfa/dataops/catalog.py

Comment thread cfa/dataops/catalog.py

Comment thread cfa/dataops/catalog.py Outdated

Comment thread cfa/dataops/catalog.py Outdated

Comment thread cfa/dataops/catalog.py

ryanraaschCDC and others added 4 commits March 25, 2026 15:56

Update cfa/dataops/catalog.py

a0db602

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update cfa/dataops/catalog.py

cbda46c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

change print to valuerror

eaa1758

update test

e65cc19

damonbayer reviewed Mar 27, 2026

View reviewed changes

Comment thread tests/test_datasets_catalog.py

add pl lazy test

e30072c

ryanraaschCDC requested a review from damonbayer March 31, 2026 13:23

damonbayer approved these changes Mar 31, 2026

View reviewed changes

ryanraaschCDC marked this pull request as ready for review March 31, 2026 16:11

damonbayer mentioned this pull request Mar 31, 2026

add lazy option to cfa.stf.data CDCgov/cfa-stf-routine-forecasting#990

Closed

ryanraaschCDC added 2 commits March 31, 2026 18:09

Merge branch 'main' of https://github.com/CDCgov/cfa-dataops into rr-…

96409e4

…61-polarslazyframe

update changelog, pyproject

9134d1c

ryanraaschCDC merged commit fd81ce7 into main Mar 31, 2026
4 checks passed

Copilot AI mentioned this pull request Apr 16, 2026

Add lazy option to cfa.stf.data functions CDCgov/cfa-stf-routine-forecasting#1029

Merged

7 tasks

Uh oh!

Conversation

ryanraaschCDC commented Mar 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

damonbayer Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryanraaschCDC commented Mar 27, 2026

Uh oh!

Uh oh!

damonbayer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants