Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to stream based API #399

Open
wants to merge 27 commits into
base: master
from

Conversation

Projects
None yet
9 participants
@LinusU
Copy link
Member

commented Sep 29, 2016

As have been discussed in a few other issues, I thought it would be interested to see what the api would look like if Multer just gave you .stream on the uploaded files. This removes the concept of storage engines completely and you can use any other module that works with streams, instead of multer-specific modules.

I would love some thoughts and feedback on this ❤️

ping @expressjs/multer-collaborators, @wesleytodd

@wesleytodd

This comment has been minimized.

Copy link
Member

commented Sep 29, 2016

Love this idea. It is much simpler, more modular, and would solve way more use cases than the current situation (#356 for example). I don't have time now to read all the code, but in theory this has my thumbs up. I will try to look through it more throughly this weekend.

LinusU added some commits Sep 30, 2016

@LinusU LinusU changed the title [WIP] Switch to stream based API Switch to stream based API Oct 1, 2016

@niftylettuce

This comment has been minimized.

Copy link
Member

commented Oct 1, 2016

My only issue @LinusU is to fix it so that you expose _makeMiddleware again because it's needed in koa-multer here https://github.com/koa-modules/multer/blob/master/index.js#L19. Here was the commit it was removed in 42a9e3b.

@LinusU

This comment has been minimized.

Copy link
Member Author

commented Oct 1, 2016

Just released 2.0.0-alpha.1 for you guys to try it out. It's published under the next-tag and thus won't be automatically installed for anyone.

🎉

@LinusU

This comment has been minimized.

Copy link
Member Author

commented Oct 1, 2016

@niftylettuce I'll look into the koa-issue shortly 👌

@LinusU

This comment has been minimized.

Copy link
Member Author

commented Oct 1, 2016

@niftylettuce

This comment has been minimized.

Copy link
Member

commented Oct 1, 2016

You are a rockstar. How do I donate to you?

On Oct 1, 2016 10:02 AM, "Linus Unnebäck" notifications@github.com wrote:

@niftylettuce https://github.com/niftylettuce see koa-modules/multer#11
koa-modules/multer#11


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#399 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAf7hWB-603_neTY5DRoVIaJ8iV_WiTdks5qvmfagaJpZM4KJnV5
.

@niftylettuce

This comment has been minimized.

Copy link
Member

commented Oct 2, 2016

Also @LinusU - is limits option still enforced?

@LinusU

This comment has been minimized.

Copy link
Member Author

commented Oct 2, 2016

How do I donate to you?

There are others who need it more than I, e.g. https://my.charitywater.org/donate/home

is limits option still enforced?

Yes they are 👌

@LinusU

This comment has been minimized.

Copy link
Member Author

commented Oct 2, 2016

2.0.0-alpha.2 is out with improvements to the error codes. Happy coding 🎉

To try it out, use the following command: npm install --save multer@next

@hacksparrow

This comment has been minimized.

Copy link
Member

commented Oct 2, 2016

👍🎉👍
On Sun, 2 Oct 2016 at 10:44 PM, Linus Unnebäck notifications@github.com
wrote:

2.0.0-alpha.2 is out with improvements to the error codes. Happy coding 🎉

To try it out, use the following command: npm install --save multer@next


You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
#399 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA5_YBKGjMMJ9UAM2cWfbp0LBglL5mHSks5qv-Z_gaJpZM4KJnV5
.

@niftylettuce

This comment has been minimized.

Copy link
Member

commented Oct 3, 2016

It's working great for me! I have a fork of koa-multer with this version added and @LinusU fork fix https://github.com/niftylettuce/koa-multer/tree/lu-fix

@LinusU LinusU referenced this pull request Oct 20, 2016

Closed

Silently ignore files #409

@qwelias

This comment has been minimized.

Copy link

commented Oct 20, 2016

Solves many problems, thanks!

But I don't understand why no encoding and mimetype properties in a file? I think it's critical.

@qwelias

This comment has been minimized.

Copy link

commented Oct 20, 2016

Oh nevermind, mime type is usually determined by file's extension, and file encoding would not help at anything. Correct if wrong.

@LinusU

This comment has been minimized.

Copy link
Member Author

commented Oct 20, 2016

That is correct, as I have understood it the encoding is only how it's sent over the wire and doesn't matter at all for the end user

var stream = fs.createReadStream(file.path)

stream.on('end', function () {
fs.unlink(file.path, function () {})

This comment has been minimized.

Copy link
@LinusU

LinusU Dec 22, 2016

Author Member

Actually, we can unlink this file as soon as it has been opened, that way the file wills still be removed even if it isn't consumed. More info: http://stackoverflow.com/a/3181726/148072

@oshalygin

This comment has been minimized.

Copy link

commented Mar 26, 2017

@wesleytodd awesome thanks I'm going to give that a whirl.

@orenklein

This comment has been minimized.

Copy link

commented Apr 3, 2017

@LinusU I like this idea of switching multer API to stream. :)

Few thoughts regarding multer new api implementation:
Are you creating a temporary file for each file field and then creating read stream from that file? (https://github.com/expressjs/multer/blob/explore-new-api/lib/read-body.js#L71)
If so, couldn't we get the same outcome with current multer using DiskStorage - createReadStream and then remove/unlink created file?
(https://github.com/expressjs/multer/blob/explore-new-api/lib/middleware.js#L25)
Furthermore, what if someone would prefer to stream/pipe uploaded file through memory instead of a temporary file?

Last one: what are your thoughts about an option to pipe the request's stream file directly?
(Without writing anything to a temporary file or memory, although I guess that's not the purpose of this library).

@LinusU

This comment has been minimized.

Copy link
Member Author

commented Aug 5, 2017

Hi @orenklein, sorry for the late reply :)

Are you creating a temporary file for each file field and then creating read stream from that file? If so, couldn't we get the same outcome with current multer using DiskStorage - createReadStream and then remove/unlink created file?

That is probably doable, although here it would be done automatically for the user...

Furthermore, what if someone would prefer to stream/pipe uploaded file through memory instead of a temporary file?

My plan is to let this decide itself, so that small files only ends up in memory, and large files on disk. Since the new api completely hides this implementation detail, it could be added at a later date without anything needing to be changed for the user!

Last one: what are your thoughts about an option to pipe the request's stream file directly?
(Without writing anything to a temporary file or memory, although I guess that's not the purpose of this library).

I think that this use case is better served by busboy, or by having a lower level api that we in itself build Multer on top of

@niftylettuce

This comment has been minimized.

Copy link
Member

commented Nov 25, 2017

@LinusU Can you publish a new package with this?

@niftylettuce

This comment has been minimized.

Copy link
Member

commented Nov 25, 2017

Actually I mean - why don't we do this directly in a new major version of multer?

@LinusU

This comment has been minimized.

Copy link
Member Author

commented Nov 27, 2017

It's currently published as multer@2.0.0-alpha.6, the plan is to make it 2.0.0 as soon as I feel comfortable with the final api :)

@LinusU LinusU referenced this pull request Feb 1, 2018

Closed

Mime type is not valid #555

@glortho

This comment has been minimized.

Copy link

commented Mar 14, 2018

Great to see this! Any forecast when multer 2.0.0 will come out of alpha?

@LinusU

This comment has been minimized.

Copy link
Member Author

commented Aug 6, 2018

Hey 👋

Really hoping to get to this soon, but unfortunately, I only have so much time to spend on open source.

Trying it out and reporting any issues (or even better, reporting that it works bug free ☺️) here would absolutely help to get me more comfortable to release it out of alpha. I don't currently feel that there is anything missing that needs to be added, just need to feel comfortable to push it out to all users ☺️

@jmonfort

This comment has been minimized.

Copy link

commented Aug 6, 2018

I've just tested and it works fine. Anxious to use it.
The few changes that caught me by surprised were:

  • File.originalname => File.originalName
  • File.mimetype => File.detectedMimeType
  • File.buffer is no longer available.

Otherwise, it's working fine.
Thanks for the reply

LinusU added some commits May 3, 2019

@LinusU

This comment has been minimized.

Copy link
Member Author

commented May 3, 2019

Published 2.0.0-alpha.7 that drops support for Node.js <6, and bumps all dependencies! 🚀

LinusU added some commits May 3, 2019

@niftylettuce

This comment has been minimized.

Copy link
Member

commented May 3, 2019

Wooooot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.