Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Apify.pushItem() should split larger arrays to fit under 9MB per API call #70

Closed
jancurn opened this issue Apr 13, 2018 · 4 comments
Closed

Apify.pushItem() should split larger arrays to fit under 9MB per API call #70

jancurn opened this issue Apr 13, 2018 · 4 comments
Labels
feature Issues that represent new features or improvements to existing features.

Comments

@jancurn
Copy link
Member

jancurn commented Apr 13, 2018 •
edited

If the array stringified to JSON is larger than 9MB, we should split it in two halfs and then recursively call pushItem for each of the halfs of the array. If it's only a single item and is larger than 9MB, then just throw an error.
@mtrunkat mtrunkat added the feature Issues that represent new features or improvements to existing features. label Jun 7, 2018
@jancurn jancurn changed the title Apify.pushData() should split larger arrays to fit under 9MB per API call Apify.pushItem() should split larger arrays to fit under 9MB per API call Jul 10, 2018
@jancurn
Copy link
Member Author

jancurn commented Jul 10, 2018

Actually we should really implement this

@mtrunkat
Copy link
Member

Lets make some function in apify-shared:

stringifyGzipChunkArray(array, maxChunkSizeBytes)

that returns an array of gziped chunks or throw an error if it's not possible because some object is larger than 9MB after compression.

@mtrunkat
Copy link
Member

And this splitting belongs to apify-client package

This was referenced Jul 26, 2018
Closed
Merged
@jancurn
Copy link
Member Author

jancurn commented Aug 22, 2018

Done

@jancurn jancurn closed this as completed Aug 22, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature Issues that represent new features or improvements to existing features.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mtrunkat @jancurn