Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Chunked file uploading #87

Closed
RyoshiKayo opened this Issue Mar 26, 2018 · 4 comments

Comments

Projects
None yet
4 participants
@RyoshiKayo
Copy link
Member

RyoshiKayo commented Mar 26, 2018

Add chunked file uploading to get around CF Size limits and such.

Ref:
-http://www.componentix.com/blog/13/file-uploads-using-nodejs-once-again

#enhancement

@BobbyWibowo

This comment has been minimized.

Copy link
Contributor

BobbyWibowo commented Mar 27, 2018

I got this working just a while ago (at least from what I can see when testing +300MB file chunked into 4 files, 100MB max). The codes are still rather dirty tho. Many repeats here and there.

My current mechanism is using Dropzone 5.2.0. They added chunked uploads support there.
Lolisafe is currently using 4.3.0 btw.

If they are chunked uploads, they will temporarily be stored at:

uploadDir/chunks/UUID/index.extension

for example:

uploadDir/chunks/123e4567-e89b-12d3-a456-426655440000/0.zip
uploadDir/chunks/123e4567-e89b-12d3-a456-426655440000/1.zip
...
uploadDir/chunks/123e4567-e89b-12d3-a456-426655440000/n.zip

then once all uploads are successfully finished, it will then POST the UUID (and original filename, file size and mime type for db) to a new API route api/upload/finishchunks so that it can finally combine then delete all the chunks and write the new file to db.

Oh, about the UUIDs, I think they're probably generated by Dropzone. It's always unique even when using the exact same file so I guess that works.

I had to add rimraf to the dependencies for deleting the temporary UUID dirs and the chunks in them btw (if someone have a native solution with fs-only or something, then please).

How do they all sound? I'll need a few days before I can do a PR though.

Missing (which I can't really be bothered to think about):

  • Handling cases like user never sending a POST request to the finishchunks API even after some or all of the chunks have been uploaded.
  • Tying the UUIDs to IP/token so that some random users can't send a POST request to the finishchunks API using those UUIDs, somehow.
@iCrawl

This comment has been minimized.

Copy link
Contributor

iCrawl commented Mar 28, 2018

@BobbyWibowo Sure man, just shoot the PR and I can go over it if you want

BobbyWibowo referenced this issue in BobbyWibowo/lolisafe Mar 28, 2018

Updates (YAY, CHUNKED UPLOADS!)
* Added new dependency: rimraf. This will be used by chunked upload support to bulk delete temporary chunk files.

* Added chunked uploads support :3

* Updated Dropzone to 5.2.0.

* More improvements to thumbnail view. Delete button will now only appear on hover. Some other details, such as file name, size and album/owner will also appear on hover. Touch devices will have all of those appear always visible by default.

* Image thumbnails will now appear on home page after successful uploads (only for WEBP, JPG, JPEG, BMP, GIF and PNG files). WEBP may not work properly in Firefox though.

* Refactored home.js to use const/let and some other stuff.

* Refactored album view. It will now display properly on mobile screen. Download Album button will also no longer be located at the top right, but right below the subtitle.

* Updated some version strings.

* And maybe some others that I can't remember.
@BobbyWibowo

This comment has been minimized.

Copy link
Contributor

BobbyWibowo commented Apr 5, 2018

been testing and perfecting this for a week. it's working well so far, but i'd like to wait for a proper eslint config or something before making a pr tailored for master 😢

@Pitu

This comment has been minimized.

Copy link
Member

Pitu commented May 21, 2018

This is already added and working on the next version releasing next week

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.