[upload] browser crashes while trying to upload large file (~600Mb)

[mariorodeghiero]

Browser crashes while trying to upload a large CSV file (~600Mb) for the resource

Acceptance

• Generate hash MD5 and SHA256 for large files

Tasks

• slipt the file in chunks and generate hash chunk by chunk and after compile in the final hash

Analysis

There are two options:

1. Uploading streams - Problem: stream upload is not supported by old browsers, since we should use Fetch API
   [Udpates]: This is not applicable. We should use FormData() and the file stream appened on it which doens't meet the way it suppoused to be done for upload file to the storage. Currently we pass a buffer, not a FormData object having a bufer/file/stream, this won't be handled by the storage service.

2. Multipart - Problem: We can directly modify queries etc of upload storage request, however this depends on which storage we use, for each provider we have to make custom changes, which will basically endup having our own Uppy plugin.

[kmanaseryan]

@mariorodeghiero can we close this? Seems has been solved in #90?

[mariorodeghiero]

I think yes.
FIXED. I hope the hash issue should be fixed in #90

[mbeilin]

@mariorodeghiero after some tests in NHS, the following error is throwing trying to upload the file ~600Mb:

[mariorodeghiero]

@mbeilin thanks for letting us know. Probably because of the file.row() and file.addSchema() not support it.

cc: @risenW

[mbeilin]

@risenW @mariorodeghiero I'm using this file, you can try this also - just download and exract the csv file from archive:
https://storage.googleapis.com/datopian-test/anon7762d88b53b9cae7c11295134f6fcb9284b81af1/EPD_202008-000000000000.csv.gz

LMK if it works for you locally.

[risenW]

@mbeilin Sorry about my earlier comment, I was using a smaller file instead. Currently working on the issue now.

[risenW]

@mbeilin This issue has been fixed and merged with the master branch. You can try it from your end.

cc: @mariorodeghiero

[mariorodeghiero]

great, let's test it.

[risenW]

@mbeilin we found the cause of the problem. It's because of the buffer method. We'll work on it.

[rufuspollock]

@risenW can we also get a reasonable local (automated) test with a large file so that we increase likelihood of finding issues here rather than when integrating into upstream code? (NB: the large file might need to be stored outside of git ...)

[risenW]

@rufuspollock Yes, I'll set this up locally.

[kmanaseryan]

How we can test this in upload large file function? In #96 it's not described

[rufuspollock]

@kmanaseryan i don't quite get

[Udpates]: This is not applicable. We should use FormData() and the file stream appened on it which doens't meet the way it suppoused to be done for upload file to the storage. Currently we pass a buffer, not a FormData object having a buffer/file/stream, this won't be handled by the storage service.

what does that mean? Why does this mean it won't be handled by the storage service. could you give a bit more detail.

[shevron]

I am a bit lost here, but here is what is needed in order to support large file uploads to Git LFS based storage as is implemented by ckan-client-js:

• We need to be able to read files (in the browser, but it would be useful to also have it Node.js) not entirely into memory but as a stream, e.g. read 1mb and process it iteratively
• We need this iterative / stream method to be used when calculating digest (sha256, md5, etc.)
• We need this iterative / stream method to be available as an input to Axios data parameter when doing POST or PUT requests. This should be possible from reading the Axios API (see below)
• We need to be able to read the file between specific locations (or seek to a specific location and then read N bytes from it)
• We need to be able to calculate a digest (sha256, md5) of a part of the file between two locations

Here is what axios accepts as upload data:

  // `data` is the data to be sent as the request body
  // Only applicable for request methods 'PUT', 'POST', 'DELETE , and 'PATCH'
  // When no `transformRequest` is set, must be of one of the following types:
  // - string, plain object, ArrayBuffer, ArrayBufferView, URLSearchParams
  // - Browser only: FormData, File, Blob
  // - Node only: Stream, Buffer

I'm really not much of a JS expect, so I can't say which one works best for us (maybe a different type depending on node vs browser) but I would be happy to jump on a call to figure this out.

[shevron]

Ok, after reading @kmanaseryan's analysis again I understand that some of these will not be supported in older browsers (IE 11 for example, and from what I can see even modern versions of Safari may be problematic but I'm not 100% sure on this one). In this case, if we need to support these browsers, the only option I see going forward is that uploading large files will only be possible with the multipart-basic Git LFS transfer mode. This can work as long as we can still calculate the sha256 digest of the entire file in the browser.

[kmanaseryan]

@shevron thanks for the insights.

• We need to be able to read files (in the browser, but it would be useful to also have it Node.js) not entirely into memory but as a stream, e.g. read 1mb and process it iteratively

This is already implemented both for node and browser. We have stream() which does that job. Even more for browser we have Node like stream, thanks to the recent updates that @risenW made

• We need this iterative / stream method to be used when calculating digest (sha256, md5, etc.)

So after the recent updates now hash() function uses streams.

• We need this iterative / stream method to be available as an input to Axios data parameter when doing POST or PUT requests. This should be possible from reading the Axios API (see below)
• We need to be able to read the file between specific locations (or seek to a specific location and then read N bytes from it)
• We need to be able to calculate a digest (sha256, md5) of a part of the file between two locations

We can implement a new method which will do this behind the scene using streams.

Here is what axios accepts as upload data:

  // `data` is the data to be sent as the request body
  // Only applicable for request methods 'PUT', 'POST', 'DELETE , and 'PATCH'
  // When no `transformRequest` is set, must be of one of the following types:
  // - string, plain object, ArrayBuffer, ArrayBufferView, URLSearchParams
  // - Browser only: FormData, File, Blob
  // - Node only: Stream, Buffer

I'm really not much of a JS expect, so I can't say which one works best for us (maybe a different type depending on node vs browser) but I would be happy to jump on a call to figure this out.

As for Axios, in the browser I've used FormData and File as data for the payload, but it didn't work, i.e the HTTP request for uploading to blob storage didn't failed, but the verification failed. In node haven't tried.

[kmanaseryan]

Closing this, you can follow datopian/ckan-client-js#32