Skip to content

feat: make preview input loading lightweight#116

Open
memadi-nv wants to merge 1 commit intomainfrom
memadi/feature/lightweight-preview
Open

feat: make preview input loading lightweight#116
memadi-nv wants to merge 1 commit intomainfrom
memadi/feature/lightweight-preview

Conversation

@memadi-nv
Copy link
Copy Markdown
Contributor

@memadi-nv memadi-nv commented Apr 10, 2026

Summary

  • Move row-limiting (nrows) from _run_internal into read_input so that
    preview() reads only the requested number of rows from disk instead of
    loading the full dataset and truncating afterward. For CSV this uses
    pd.read_csv(nrows=…) natively; for Parquet it applies head() after load.
  • Simplify _run_internal by removing the preview_num_records parameter that
    was threaded through detection, replacement, and rewrite workflows — the
    dataframe is now already the right size on entry.
  • Move the "Loaded N records" log into reader.py (with a "Preview mode"
    prefix when nrows is set).
  • Add unit tests for nrows behavior: CSV truncation, Parquet truncation,
    nrows > len, nrows=None, attribute preservation, and remote-CSV
    forwarding. Update logging assertions to match the new log format.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Refactoring

Testing

  • make test passes locally
  • make check passes locally (format + lint + typecheck + lock-check)
  • Added/updated tests for changes

Documentation

  • If docs changed: make docs-build passes locally

Related Issues

Signed-off-by: memadi <memadi@nvidia.com>
@memadi-nv memadi-nv requested a review from a team as a code owner April 10, 2026 23:24
@memadi-nv memadi-nv changed the title feature: make preview input loading lightweight feat: make preview input loading lightweight Apr 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant