Problem in very large file upload

[ascaler]

Most of the time when upload size is under 1M it upload correctly and we get all the data in request.req but if we try to upload 100MB file with mode: stream and multipart : stream , in that case it only give 65K data and can't find a way to get more data. Is that a know issue or is there any trick to get remaining data? Any help in this regard is highly appreciated.

[hueniverse]

Did you change server config payload.maxBytes?

https://github.com/spumko/hapi/blob/master/docs/Reference.md#server.config.payload

[ascaler]

Thanks for the quick reply. Yes i did setup the large maxBytes and there are two issues in doing that. We wanted to setup for 5GB size for upload but that doesn't work as it seems it allocate large memory. If I setup like 10MB then from client side I can see that we are sending about stream with expect continue and as long as server keep sending response, client can send more data in chunk of large data size. But in this case the buffer still show only 64KB of data. I was expecting some kind of stream implementation or some kind of event that says receive this data and more is in queue till client sent all the data. Our goal is to receive very large file as S3 does using multipart upload.

[hueniverse]

Sorry, maxBytes is not relevant here as you are not setting payload to raw or parse. It has no effect for payload mode stream. Can you show me what your route config looks like?

[ascaler]

var ProcessPUT = {
payload: {
maxBytes:10485760,
mode: 'stream',
multipart:'stream'
},
handler: function (request, next) {
// All the processing is in there

   // For the uploading request we are expecting 100-continue 
   if(request.headers.expect === '100-continue')
    {
        console.log('In 100 continue');
        console.log(request.headers['content-length']);
        console.log(request.stream);
        console.log(request.raw.req._readableState);
     }
}

}

output of this comes out like -

In 100 continue
18171420
undefined
{ highWaterMark: 16384,
buffer: [],
length: 0,
pipes: null,
pipesCount: 0,
flowing: false,
ended: false,
endEmitted: false,
reading: false,
calledRead: false,
sync: true,
needReadable: false,
emittedReadable: false,
readableListening: false,
objectMode: false,
defaultEncoding: 'utf8',
ranOut: false,
awaitDrain: 0,
readingMore: false,
decoder: null,
encoding: null }

Once we check Buffer then size of buffer comes out to be 64K only.

[hueniverse]

Can you try this again with master? Note that 2.0 is very different in how payload is configured but I completely rewrote payload handling.

[ascaler]

Thank you Eran, I am going to try that and will let you know. Is that any example or update in document to know new process... hope its there and will figure it out. Thanks again.

[hueniverse]

@ascaler The docs/Reference.md is up to date.

[hueniverse]

Assuming it's fixed.

[dheerajsingh25]

Can you please provide me a working example to upload large file(as multipart) with expect-100 header?