Error when attempting to upload a file that exceeds 'maxBytes' limit

[koukuko]

Error when attempting to upload a file that exceeds 'maxBytes' limit, is there anything wrong with skipper or is it my problem?

E:\h.acfun.tv.admin\node_modules\sails\node_modules\skipper\node_modules\skipper-disk\standalone\build-progress-stream.js:129
 Garbage collecting file `' + __newFile.filename + '` located @ ' + fd + '...'
                                                                    ^
ReferenceError: fd is not defined
    at gc (E:\h.acfun.tv.admin\node_modules\sails\node_modules\skipper\node_modules\skipper-disk\standalone\build-progress-stream.js:129:96)
    at Transform.singleFileProgress (E:\h.acfun.tv.admin\node_modules\sails\node_modules\skipper\node_modules\skipper-disk\standalone\build-progress-stream.js:134:9)
    at Transform.emit (events.js:95:17)
    at Transform.__progress__._transform (E:\h.acfun.tv.admin\node_modules\sails\node_modules\skipper\node_modules\skipper-disk\standalone\build-progress-stream.js:42:10)
    at Transform._read (_stream_transform.js:179:10)
    at Transform.Readable.read (_stream_readable.js:323:10)
    at flow (_stream_readable.js:591:52)
    at WriteStream.<anonymous> (_stream_readable.js:575:7)
    at WriteStream.emit (events.js:92:17)
    at onwriteDrain (_stream_writable.js:289:12)
    at afterWrite (_stream_writable.js:277:5)

[Khufu-I]

I'm also experiencing this problem. My sample code is

var uploadOptions = {
    dirname: 'D:/Khufu_I/Videos',
   maxBytes: 20 * 1000 * 1000
}

req.file('video').upload(uploadOptions, function videoUploaded(err, uploadedFiles) {
  if (err) return res.serverError(err);
  if (!uploadedFiles || !uploadedFiles[0]) {
    return res.badRequest('Video was not uploaded.');
  }
});

The program crashes without hitting any of the errors.

[ytanay]

As far as I can see, this seems to be a issue with the disk transport, and not with skipper itself. In any case, I can offer a temporary and extremely dirty solution if anyone is interested.

There is an interesting bug with skipper when a file which exceeds the maxBytes property is uploaded. Sometimes it is stored, and other times simply crashes the server. What I currently do is read the bytesExpected property from the incoming request fileparser, and work according to that.

if(req._fileparser.form.bytesExpected > 10000000){
  req.connection.destroy();
  return res.end('File exceeds maxSize. Aborting.'); // This doesn't actually get sent, so you can skip this line.
}

req.file('field').upload({
  maxBytes: Number.MAX_VALUE
}, function(err, files){
  //file was uploaded?
});

I know, it's horrible, but it actually works quite well for me. As the connection gets destroyed before the upload begins, it's fine for AJAX requests, but not so great for regular form based uploads.

[billyshena]

I'm encoutering the exact same issue, any fixes yet?

[dzcpy]

I've created a PR here balderdashy/skipper-disk#6 to fix this issue
Please help to test

[lyhoanglong]

anyone fixed this problem?

[dzcpy]

@lyhoanglong please check my last comment for the fix. It works for me but more tests are welcome to make sure the fix is reliable. Thanks!

[lyhoanglong]

@andyhu I'll check this, thanks for reply.

[boogerlad]

When I upload a file significantly larger than maxBytes(3x), a response cannot get sent.

        req
        .file('image')
        .upload
        (
            {
                saveAs: req.params.id,
                maxBytes: 5242880
            },
            function(err, files)
            {
                console.log("I'm getting here")
                if(err)
                {
                    res.serverError(err);
                }
...

in the console I get

I'm getting here
error: Sending 500 ("Server Error") response: 
 Upload Error: Upload limit of 5242880 bytes exceeded (5274782 bytes written)
    at Transform.singleFileProgress (/root/cutequickposweb/node_modules/sails/node_modules/skipper/node_modules/skipper-disk/standalone/build-progress-stream.js:113:17)
    at Transform.emit (events.js:95:17)
    at Transform.__progress__._transform (/root/cutequickposweb/node_modules/sails/node_modules/skipper/node_modules/skipper-disk/standalone/build-progress-stream.js:42:10)
    at Transform._read (_stream_transform.js:179:10)
    at Transform.Readable.read (_stream_readable.js:340:10)
    at flow (_stream_readable.js:607:52)
    at WriteStream. (_stream_readable.js:591:7)
    at WriteStream.emit (events.js:92:17)
    at onwriteDrain (_stream_writable.js:288:12)
    at afterWrite (_stream_writable.js:276:5)
    at onwrite (_stream_writable.js:269:7) { [Upload Error: Upload limit of 5242880 bytes exceeded (5274782 bytes written)]
  code: 'E_EXCEEDS_UPLOAD_LIMIT',
  name: 'Upload Error',
  maxBytes: 5242880,
  written: 5274782,
  message: 'Upload limit of 5242880 bytes exceeded (5274782 bytes written)' }

A file barely above the limit allows a response to be sent.

[dalton5]

Hi,

I still have the error.

I have sails js with skipper 0.5.5 and skipper-disk 0.5.4.

I try to upload a csv of 45 MB.

The error is raised:

var outs__ = fsx.createWriteStream(__newFile.fd, encoding);

        // When the file is done writing, call the callback
        outs__.on('finish', function successfullyWroteFile() {
            log('finished file: ' + __newFile.filename);
            done();
        });
        outs__.on('E_EXCEEDS_UPLOAD_LIMIT', function (err) {
            done(err);
        });

The detail is:

code: "E_EXCEEDS_UPLOAD_LIMIT"
maxBytes: 15000000
message: "Upload limit of 15000000 bytes exceeded (15048979 bytes written)"
name: "Upload Error"
stack: undefined
written: 15048979

An idea on the fix?

Thanks,

[sailsbot]

Thanks for posting, @koukuko. I'm a repo bot-- nice to meet you!

It has been 60 days since there have been any updates or new comments on this page. If this issue has been resolved, feel free to disregard the rest of this message. On the other hand, if you are still waiting on a patch, please:

• review our contribution guide to make sure this submission meets our criteria (only verified bugs with documented features, please; no questions, commentary, or bug reports about undocumented features or unofficial plugins)
• create a new issue with the latest information, including updated version details with error messages, failing tests, and a link back to the original issue. This allows GitHub to automatically create a back-reference for future visitors arriving from search engines.

Thanks so much for your help!

[RWOverdijk]

Bump. How to fix this?

[sailsbot]

Thanks for posting, @koukuko. I'm a repo bot-- nice to meet you!

It has been 30 days since there have been any updates or new comments on this page. If this issue has been resolved, feel free to disregard the rest of this message. On the other hand, if you are still waiting on a patch, please:

• review our contribution guide to make sure this submission meets our criteria (only verified bugs with documented features, please; no questions, commentary, or bug reports about undocumented features or unofficial plugins)
• create a new issue with the latest information, including updated version details with error messages, failing tests, and a link back to the original issue. This allows GitHub to automatically create a back-reference for future visitors arriving from search engines.

Thanks so much for your help!

[wasbridge]

I have the same problem as @ameilland how can I make it so that skipper gives me the error so that I can inform the user?

[trungducng]

Still error till now
Any fix please?

I'm working on sails ver 0.12.14

[cubetechnology]

@trungducng just trying to fix this i put in ur upload options maxBytes:10000000000000, and works. i just put 10000000000000 i think if u need < than that should work. let me know if u solved with this

[nasiruddin-saiyed]

@cubetechnology @dzcpy got this error while uploading file over stream any solution ?? need a robust solution for dynamic up and down streaming over fs to store file via incoming request.

Here is the example of router hook i am trying to upload file

const intervalId = setInterval(async function () {
     if (req._fileparser.closed) {
         clearInterval(intervalId);
         fileObjects = await tasks;
         var allUploads = fileObjects;
         var mainUpload = {};
         allUploads.map(function (upsetram) {
             return upsetram._files.map(function (fl) {
                 if (fl.stream.fd) {
                     fs. createReadStream(fl.stream.fd) //.pipe(
                     let file = {
                         name: fl.stream.name,
                         filename: fl.stream.filename,
                         byteOffset: fl.stream.byteOffset,
                         byteCount: fl.stream.byteCount,
                         field: fl.stream.field,
                         fd: fl.stream.fd
                     }
                     if (mainUpload[fl.stream.name]) {
                         mainUpload[fl.stream.name].push(file);
                     } else {
                         Object.assign(mainUpload, {
                             [fl.stream.name]: [file]
                         });
                     }
                     console.log(file);
                 }
                 return fl;
             });
         });
         console.log(mainUpload);
         Object.assign(req.body, mainUpload);
         delete req._fileparser
         return next();
     } else if (new Date().getTime() - startTime > TIMEOUT) {
         clearInterval(intervalId);
         new Error("Timeout triggered before form closed");
         return next();
     }
 }, 100);