Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Upload of large files is failing #14

Closed
2 tasks
Daniellappv opened this issue Nov 17, 2020 · 3 comments · Fixed by #17
Closed
2 tasks

Data Upload of large files is failing #14

Daniellappv opened this issue Nov 17, 2020 · 3 comments · Fixed by #17
Assignees

Comments

@Daniellappv
Copy link
Member

Daniellappv commented Nov 17, 2020

Size of NHS files vary between 5.8 GB and 6.3 GB, Avg is roughly 6.1 GB

We tested uploading a 6 GB file via the GUI but it's crashing, see screenshot

image

Another test with a 6.3 GB file // with Network requests/responses and errors when uploading

image

10 mins later

image

20 mins later

image

Error: "Paused before potential out-of-memory crash"

Acceptance

  • Upload via GUI should work for their max-sized files (6.3 GB)
  • Shouldn’t be slower than the current solution (uploading files manually to the bucket)
@mariorodeghiero
Copy link
Contributor

Just a note:

  • this failed before the upload request, I think is about generating a hash or some step after this.

@mariorodeghiero
Copy link
Contributor

@Daniellappv how the root cause is in datapub repository I opened the issue there. You can check here datopian/datapub#78

@risenW
Copy link
Contributor

risenW commented Nov 25, 2020

@Daniellappv how the root cause is in datapub repository I opened the issue there. You can check here datopian/datapub#78

@mariorodeghiero The cause is actually coming from frictionless.js. Check out this pull here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants