Large File Handling (eg 2.5 MB file) Slower When InMemory is True

[jpfluger]

When inMemory is true and file.buffer is larger than 64 Kb, there is an "issue" that I don't think is fixable right now b/c it's the way Node.js currently allocates memory for large buffers. (see StackOverflow and this blog)

In multer/master, I have mocha tests (see test/inmemory.js) that can be used to test this issue. When running, the multi-file upload test take an average of 1 second. Compare this to the write-to-disk multi-file upload taking 100 ms. In local tests, I changed-up my inMemory Buffer code to a writable stream and piped the filestream directly to it but still would get 1 second performance. I don't think my test hardware is to blame (i7-4810MQ w/ 32 Gig RAM on SSDs).

I also ran the tests against the latest node in development 0.11.14 but there's a bug preventing mocha from completing. Need to keep an eye on this b/c TJ wrote "[there's been] significant performance work done in core features (Buffers, TLS, streams)." reference

Hopefully I'm wrong with this issue and someone knowing more about how core streams/Buffers work can suggest ways to improve inMemory performance for large buffers.

[IsaiahJTurner]

Buffer is null for me. Is that normal?

[jpfluger]

If inMemory is true and the buffer is null, this is not normal. If the default setting is used (where inMemory is false), then yes it is normal. Maybe it's the latter?

[LinusU]

I'm not sure that this is an issue anymore with never node. From the blog:

UPDATE: None of this should be necessary, as FileReadStream in the latest node uses buffers by default. However, it appears that either I'm doing something wrong or the docs are out of date, as it doesn't work that way on node HEAD.

We are using buffers internally in 1.0.0.

Please reopen if you feel that this is still an issue.

[joelsouza]

I'm noticing multer is often very slower using memoryStorage. I running this app on Heroku (2 dynos 2.5GB RAM each) and can't reproduce this problem local.

I'm uploading/processing binary files < 10mb

Node: 10.13.0
Express: 4.16.4
Multer: 1.4.1

Procfile

web: node --optimize_for_size --max_old_space_size=960 --gc_interval=100 ./bin/www

From the Heroku logs:

14 Dec 2018 22:10:40.888111...app web.2 - multer: 546.008ms
14 Dec 2018 22:10:41.273111...app web.2 - multer: 799.298ms
14 Dec 2018 22:11:02.265113...app web.2 - multer: 14022.644ms
14 Dec 2018 22:11:20.837113...app web.1 - multer: 15048.781ms
14 Dec 2018 22:11:27.108113...app web.2 - multer: 15581.935ms
14 Dec 2018 22:11:50.460112...app web.2 - multer: 9385.592ms
14 Dec 2018 22:12:10.792112...app web.1 - multer: 1873.741ms
14 Dec 2018 22:12:17.253112...app web.1 - multer: 6949.123ms
14 Dec 2018 22:12:18.849111...app web.2 - multer: 459.384ms
14 Dec 2018 22:12:19.396111...app web.1 - multer: 762.943ms
14 Dec 2018 22:12:48.720112...app web.2 - multer: 8931.458ms
14 Dec 2018 22:12:48.927113...app web.2 - multer: 33099.362ms
14 Dec 2018 22:13:12.387112...app web.1 - multer: 6995.448ms
14 Dec 2018 22:13:52.009112...app web.2 - multer: 1641.400ms
14 Dec 2018 22:13:56.123111...app web.1 - multer: 448.621ms
14 Dec 2018 22:13:56.851111...app web.2 - multer: 746.916ms
14 Dec 2018 22:13:58.788112...app web.2 - multer: 7218.541ms
14 Dec 2018 22:14:22.591111...app web.2 - multer: 402.261ms
14 Dec 2018 22:14:22.840111...app web.2 - multer: 688.741ms
14 Dec 2018 22:14:23.346112...app web.1 - multer: 5194.058ms
14 Dec 2018 22:14:26.019111...app web.2 - multer: 534.709ms
14 Dec 2018 22:15:13.401112...app web.2 - multer: 3775.944ms
14 Dec 2018 22:15:17.796111...app web.2 - multer: 397.492ms
14 Dec 2018 22:15:18.204111...app web.2 - multer: 752.125ms
14 Dec 2018 22:15:41.958111...app web.1 - multer: 544.359ms
14 Dec 2018 22:15:42.480111...app web.2 - multer: 836.787ms
14 Dec 2018 22:15:47.693112...app web.1 - multer: 1727.417ms
14 Dec 2018 22:15:52.398111...app web.1 - multer: 791.530ms
14 Dec 2018 22:15:53.531112...app web.1 - multer: 6623.814ms
14 Dec 2018 22:15:56.511112...app web.1 - multer: 4379.295ms
14 Dec 2018 22:16:30.912111...app web.2 - multer: 585.118ms
14 Dec 2018 22:17:10.618112...app web.2 - multer: 1406.447ms
14 Dec 2018 22:17:13.976111...app web.2 - multer: 117.054ms
14 Dec 2018 22:17:16.091112...app web.2 - multer: 1649.331ms

My route is somthing like this:

const router = require('express').Router()
const multer = require('multer')
const upload = multer({ storage: multer.memoryStorage() })
const fileUpload = upload.single('file')

router.post('/file', (req, res) => {

  console.time('multer')
  fileUpload(req, res, async (err) => {
    console.timeEnd('multer')

    if (err)  console.error(err)
    	
    // 
    // upload file to AWS S3.
    // 

    return res.json({ })
  })
})

module.exports = router

Is there any way to search for the root of the problem?