Switch to stream based API

[LinusU]

As have been discussed in a few other issues, I thought it would be interested to see what the api would look like if Multer just gave you .stream on the uploaded files. This removes the concept of storage engines completely and you can use any other module that works with streams, instead of multer-specific modules.

I would love some thoughts and feedback on this ❤️

ping @expressjs/multer-collaborators, @wesleytodd

[wesleytodd]

Love this idea. It is much simpler, more modular, and would solve way more use cases than the current situation (#356 for example). I don't have time now to read all the code, but in theory this has my thumbs up. I will try to look through it more throughly this weekend.

[niftylettuce]

My only issue @LinusU is to fix it so that you expose _makeMiddleware again because it's needed in koa-multer here https://github.com/koa-modules/multer/blob/master/index.js#L19. Here was the commit it was removed in 42a9e3b.

[LinusU]

Just released 2.0.0-alpha.1 for you guys to try it out. It's published under the next-tag and thus won't be automatically installed for anyone.

🎉

[LinusU]

@niftylettuce I'll look into the koa-issue shortly 👌

[LinusU]

@niftylettuce see koa-modules/multer#11

[niftylettuce]

You are a rockstar. How do I donate to you?

On Oct 1, 2016 10:02 AM, "Linus Unnebäck" notifications@github.com wrote:

@niftylettuce https://github.com/niftylettuce see koa-modules/multer#11
koa-modules/multer#11

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#399 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAf7hWB-603_neTY5DRoVIaJ8iV_WiTdks5qvmfagaJpZM4KJnV5
.

[niftylettuce]

Also @LinusU - is limits option still enforced?

[LinusU]

How do I donate to you?

There are others who need it more than I, e.g. https://my.charitywater.org/donate/home

is limits option still enforced?

Yes they are 👌

[LinusU]

2.0.0-alpha.2 is out with improvements to the error codes. Happy coding 🎉

To try it out, use the following command: npm install --save multer@next

[hacksparrow]

👍🎉👍
On Sun, 2 Oct 2016 at 10:44 PM, Linus Unnebäck notifications@github.com
wrote:

2.0.0-alpha.2 is out with improvements to the error codes. Happy coding 🎉

To try it out, use the following command: npm install --save multer@next

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
#399 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA5_YBKGjMMJ9UAM2cWfbp0LBglL5mHSks5qv-Z_gaJpZM4KJnV5
.

[niftylettuce]

It's working great for me! I have a fork of koa-multer with this version added and @LinusU fork fix https://github.com/niftylettuce/koa-multer/tree/lu-fix

[qwelias]

Solves many problems, thanks!

But I don't understand why no encoding and mimetype properties in a file? I think it's critical.

[qwelias]

Oh nevermind, mime type is usually determined by file's extension, and file encoding would not help at anything. Correct if wrong.

[LinusU]

That is correct, as I have understood it the encoding is only how it's sent over the wire and doesn't matter at all for the end user

[LinusU]

Actually, we can unlink this file as soon as it has been opened, that way the file wills still be removed even if it isn't consumed. More info: http://stackoverflow.com/a/3181726/148072

[jonchurch]

It's not yet stable in Node, but WHATWG streams seem like they'd be a nice addition here eventually. That might be Multer 3.0, though!

[bwgjoseph]

Hi @LinusU,

I was testing on one of my (typescript) project, and it seem to throw some ESM error while running with the latest multer (rc3)

Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: \feathers-ottoman-demo\node_modules\multer\index.js   
require() of ES modules is not supported.
require() of \feathers-ottoman-demo\node_modules\multer\index.js from \feathers-ottoman-demo\src\services\attachments\attachments.service.ts is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
Instead rename index.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from \feathers-ottoman-demo\node_modules\multer\package.json.

    at Module._extensions..js (internal/modules/cjs/loader.js:1080:13)
    at Object.nodeDevHook [as .js] (\feathers-ottoman-demo\node_modules\ts-node-dev\lib\hook.js:63:13)
    at Module.load (internal/modules/cjs/loader.js:928:32)
    at Function.Module._load (internal/modules/cjs/loader.js:769:14)
    at Module.require (internal/modules/cjs/loader.js:952:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at Object.<anonymous> (\feathers-ottoman-demo\src\services\attachments\attachments.service.ts:6:1)
    at Module._compile (internal/modules/cjs/loader.js:1063:30)
    at Module._compile (\feathers-ottoman-demo\node_modules\source-map-support\source-map-support.js:568:25)      
    at Module.m._compile (C:\Users\Joseph\AppData\Local\Temp\ts-node-dev-hook-005731216882142842.js:69:33)
[ERROR] 14:17:26 Error: Must use import to load ES Module: \feathers-ottoman-demo\node_modules\multer\index.js    
require() of ES modules is not supported.
require() of \feathers-ottoman-demo\node_modules\multer\index.js from \feathers-ottoman-demo\src\services\attachments\attachments.service.ts is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
Instead rename index.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from \feathers-ottoman-demo\node_modules\multer\package.json.

This is my tsconfig which I added ts-node configuration

{
  "ts-node": {
    // See https://github.com/TypeStrong/ts-node/issues/922
    // these options are overrides used only by ts-node
    // same as our --compilerOptions flag and our TS_NODE_COMPILER_OPTIONS environment variable
    "compilerOptions": {
      "module": "commonjs"
    }
  },
  "compilerOptions": {
    "target": "es2018",
    "module": "commonjs",
    "outDir": "./lib",
    "rootDir": "./src",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true
  },
  "exclude": [
    "test"
  ]
}

And I'm using multer like such

import multer from 'multer';

const MAX_FILE_SIZE = 5242880;
const MAX_FILE_UPLOAD_LIMIT = 5;

// Instantiate multer
const multerware = multer({
  limits: {
    fileSize: MAX_FILE_SIZE,
    files: MAX_FILE_UPLOAD_LIMIT,
  },
});

multerware.array('files'),

If I add type: 'module in package.json, it doesn't change anything. The same error will be thrown.

I switched back to rc2 with the exact same configuration, and it runs perfectly.

I'm not familiar with the whole ESM thingy, any idea?

[LinusU]

@bwgjoseph I think that if you follow this answer it should help you get TypeScript up and running with ESM:

https://stackoverflow.com/a/67371788/148072

[bwgjoseph]

So I used this config as mentioned in the SO post you linked

"target": "ES2020",
"module": "commonjs",
"lib": ["ES2020"],
"outDir": "./lib",
"rootDir": "./src",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true

where I stick to cjs, and everything run fine without rc3. Once I switched to rc3, it throws the following error

$ npm run mocha

> feathers-ottoman-demo@0.0.0 mocha \bwgjoseph\feathers-ottoman-demo
> ts-mocha "test/**/*.ts" --recursive --exit


TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for \bwgjoseph\feathers-ottoman-demo\test\app.test.ts
    at Loader.defaultGetFormat [as _getFormat] (internal/modules/esm/get_format.js:71:15)
    at Loader.getFormat (internal/modules/esm/loader.js:102:42)
    at Loader.getModuleJob (internal/modules/esm/loader.js:231:31)
    at Loader.import (internal/modules/esm/loader.js:165:17)
    at formattedImport (\bwgjoseph\feathers-ottoman-demo\node_modules\mocha\lib\nodejs\esm-utils.js:7:14)
    at Object.exports.requireOrImport (\bwgjoseph\feathers-ottoman-demo\node_modules\mocha\lib\nodejs\esm-utils.js:48:32)
    at Object.exports.loadFilesAsync (\bwgjoseph\feathers-ottoman-demo\node_modules\mocha\lib\nodejs\esm-utils.js:88:20)
    at singleRun (\bwgjoseph\feathers-ottoman-demo\node_modules\mocha\lib\cli\run-helpers.js:125:3)
    at Object.exports.handler (\bwgjoseph\feathers-ottoman-demo\node_modules\mocha\lib\cli\run.js:374:5)

Not exactly sure why it points to mocha though maybe I ran npm run mocha command

Say that I follow what the poster mentioned and set the config to

"target": "ES2020",
"module": "ES2020",
"lib": ["ES2020"],
"outDir": "./lib",
"rootDir": "./src",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"allowSyntheticDefaultImports": true,
"moduleResolution": "node"

// and adding this to package.json
"type": "module"

It wouldn't even run in rc2

$ npm run mocha

> feathers-ottoman-demo@0.0.0 mocha bwgjoseph\feathers-ottoman-demo
> ts-mocha "test/**/*.ts" --recursive --exit


bwgjoseph\feathers-ottoman-demo\test\app.test.ts:1
import assert from 'assert';
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at wrapSafe (internal/modules/cjs/loader.js:979:16)
    at Module._compile (internal/modules/cjs/loader.js:1027:27)
    at Module.m._compile (bwgjoseph\feathers-ottoman-demo\node_modules\ts-node\src\index.ts:439:23)
    at Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
    at Object.require.extensions.<computed> [as .ts] (bwgjoseph\feathers-ottoman-demo\node_modules\ts-node\src\index.ts:442:12)
    at Module.load (internal/modules/cjs/loader.js:928:32)
    at Function.Module._load (internal/modules/cjs/loader.js:769:14)
    at Module.require (internal/modules/cjs/loader.js:952:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at Object.exports.requireOrImport (bwgjoseph\feathers-ottoman-demo\node_modules\mocha\lib\nodejs\esm-utils.js:56:20)
    at Object.exports.loadFilesAsync (bwgjoseph\feathers-ottoman-demo\node_modules\mocha\lib\nodejs\esm-utils.js:88:20)
    at singleRun (bwgjoseph\feathers-ottoman-demo\node_modules\mocha\lib\cli\run-helpers.js:125:3)
    at Object.exports.handler (bwgjoseph\feathers-ottoman-demo\node_modules\mocha\lib\cli\run.js:374:5)

Can't quite figure out the cause of it. If you think this has nothing to do with the recent change, then I'll figure out something myself. Was thinking not sure if this was caused by changes in rc3 hence I reported it here

[LinusU]

@bwgjoseph I think that your problem is generic to using any ESM module with your specific setup (which seems to be TypeScript + ts-node + ts-mocha), and I'm a bit wary of cluttering up this thread with that... That being said, I don't know of a more appropriate place for this, and I do want to help you 😄

I haven't personally used ts-mocha, but I have used ts-node with ESM packages without problems. From looking around at ts-mocha, it seems like it's a small shim around mocha and ts-node.

Could you try to switch our your ts-mocha command to something like this:

- ts-mocha "test/**/*.ts" --recursive --exit
+ node --loader ts-node/esm node_modules/.bin/mocha  "test/**/*.ts" --recursive --exit

You can find more information about this here: TypeStrong/ts-node#1007

As outlined in those step, you also need to:

• Set "module": "ESNext" or "ES2015" so that TypeScript emits import/export syntax.
• Set "type": "module" in your package.json, which is required to tell node that .js files are ESM instead of CommonJS.
• Install ts-node locally, not globally. npm install ts-node or yarn add ts-node
• Include file extensions in your import statements, or pass --experimental-specifier-resolution=node Idiomatic TypeScript should import foo.ts as import 'foo.js'; TypeScript understands this.

I believe that you have already done at least the three first steps. The fourth is admittedly a bit weird when working with TypeScript, so the easiest way to get up and running is probably to pass the --experimental-specifier-resolution=node flag:

- ts-mocha "test/**/*.ts" --recursive --exit
+ node --loader ts-node/esm --experimental-specifier-resolution=node node_modules/.bin/mocha  "test/**/*.ts" --recursive --exit

There is also more support coming with the next version of TypeScript, which is currently in beta. See the release notes here: https://devblogs.microsoft.com/typescript/announcing-typescript-4-5-beta/. Potentially that will help making this smoother as well!

---

For some background on why this change is happening, I would recommend some of these resources:

• https://blog.sindresorhus.com/get-ready-for-esm-aa53530b3f77
• https://blog.sindresorhus.com/hello-modules-d1010b4e777b
• https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c#how-can-i-move-my-commonjs-project-to-esm

[bwgjoseph]

Hey @LinusU,

Thanks for helping me out, appreciate it.

I still am not able to get it working properly even with the different trials and errors. I think I don't understand enough of all these, will do up more reading first. So I reverted back to rc2 for now.

And I also don't want to clutter this thread further more if this is not a multer issue

Cheers!

[mohe2015]

Maybe use this opportunity to update all packages to their new major versions? And maybe remove mkdirp and replace it with its native counterpart https://nodejs.org/dist/latest-v16.x/docs/api/fs.html#fs_fspromises_mkdir_path_options / https://nodejs.org/dist/latest-v16.x/docs/api/fs.html#fsmkdirpath-options-callback

Edit: e.g I did an almost working update (tests hang at the end) at https://github.com/mohe2015/multer/tree/explore-new-api-update-esm But this also includes switching to esm which may not be wanted. Also this was not tested at all except running the tests

[LinusU]

@mohe2015 I believe that this is already done in this branch? It's using ESM and the latest major version of all dependencies?

[mohe2015]

@mohe2015 I believe that this is already done in this branch? It's using ESM and the latest major version of all dependencies?

Yeah seems like I messed something up. Still the conflicts probably need to be resolved at some point but I don't know if this would be needed now.

[LinusU]

Still the conflicts probably need to be resolved at some point but I don't know if this would be needed now.

When we release version 2.0.0 I will rename the master branch to v1.x and this branch to master

[niftylettuce]

Bump 😉

[bryanph]

@LinusU How do you feel about just releasing the 2.0 and seeing how it goes? That way more people get to try it and any bugs that might exist in it will surface quicker. Also gives the community a chance to contribute if bugs are found. What do you think?

In any case thanks for this project, it's helped me out a lot! 😁