Skip to content

remove row_limit parameter, use local config.yaml instead#87

Merged
zach-maddox merged 4 commits intomainfrom
row-limit-local-only
Apr 20, 2026
Merged

remove row_limit parameter, use local config.yaml instead#87
zach-maddox merged 4 commits intomainfrom
row-limit-local-only

Conversation

@zach-maddox
Copy link
Copy Markdown
Collaborator

This fixes the problem where the default 1000 row limit is used inside Data Cloud, which requires users to explicitly pass a higher row limit in every read call.

Summary of changes:

  • Removed row_limit parameter from read_dlo/read_dmo across the public API and all reader implementations
  • Local development reads are automatically capped at 1000 rows via default_row_limit in config.yaml's reader_config.options — no user action needed
  • Deployed environments provide their own reader_config (which doesn't include default_row_limit), so no limit is applied and all records are processed
  • Users who need a specific row limit can use PySpark's native .limit() on the returned DataFrame

@salesforce-cla
Copy link
Copy Markdown

Thanks for the contribution! It looks like @zach-maddox is an internal user so signing the CLA is not required. However, we need to confirm this.

@zach-maddox zach-maddox merged commit 243fbb5 into main Apr 20, 2026
3 checks passed
@zach-maddox zach-maddox deleted the row-limit-local-only branch April 20, 2026 17:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants