Skip to content

Forbid to download a too large file#89

Merged
Pierlou merged 9 commits intomainfrom
fix/json-batchsize
Jan 9, 2026
Merged

Forbid to download a too large file#89
Pierlou merged 9 commits intomainfrom
fix/json-batchsize

Conversation

@Pierlou
Copy link
Copy Markdown
Contributor

@Pierlou Pierlou commented Jan 8, 2026

Following datagouv/data.gouv.fr#1921, we have two options:

  • fix the download (aka change how we read the chunks of data from Postgrest, which is not the same for csv and json. I have not yet found an elegant way to achieve that, but I have an ugly one)
  • forbid to download if the result's size if longer than the batch size, because such operations are costly and at some point it could be fair to ask the user to use the source file directly

This PR implements the second option, but there are valid reasons to go for the first one. Happy to discuss this!

@Pierlou Pierlou requested a review from maudetes January 8, 2026 15:55
Copy link
Copy Markdown
Contributor

@maudetes maudetes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! I'm good with this for now.

We can fix large file download when we have a good benchmark in place

Comment thread api_tabular/core/data.py Outdated
Copy link
Copy Markdown
Contributor

@maudetes maudetes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I trust you on the response refactoring 🤞

@Pierlou Pierlou merged commit f12e2f2 into main Jan 9, 2026
4 checks passed
@Pierlou Pierlou deleted the fix/json-batchsize branch January 9, 2026 13:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants