Skip to content

Batch polars parquet uploads#1268

Closed
Reichenbachian wants to merge 2 commits intomainfrom
claude/chunky_parquet_salsa
Closed

Batch polars parquet uploads#1268
Reichenbachian wants to merge 2 commits intomainfrom
claude/chunky_parquet_salsa

Conversation

@Reichenbachian
Copy link
Copy Markdown
Contributor

@Reichenbachian Reichenbachian commented Feb 2, 2026

Original Prompt

clone structify-python and then batch upload_df to 10k at a time? and make a pr into main for it?

Surprises and Discoveries

  • None.

Changes Made

  • Batch _upload_df uploads in 10k-row slices with an in-function constant.
  • Apply batching for all callers (tag, match, enhance, etc.).

Tests

Not run (not requested).

@Reichenbachian Reichenbachian marked this pull request as ready for review February 2, 2026 20:02
Copy link
Copy Markdown
Collaborator

@falafel2222 falafel2222 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants