Stream the http response

[bajtos]

Consider a remotable method that returns a stream as the result, for example "download file" in loopback-storage-service (source).

The current implementation takes the response object as an argument and does not call the provided callback. This means that "after" hooks are never called. (remoting metadata).

File.download = function(file, res, cb) {
  var reader = fs.createReadStream('storage/' + file);
  reader.on('error', function (err) {
    res.type('application/json');
    res.send(500, { error: err });
  });
  reader.pipe(res);
  // callback is intentionally not invoked
};

File.remoteMethod('download', {
  accepts: [
    {arg: 'file', type: 'string', 'http': {source: 'path'}},
    {arg: 'res', type: 'object', 'http': {source: 'res'}}
  ],
  http: {
    verb: 'get',
    path: '/download/:file'
  }
});

We should make streams a first-class citizen in strong-remoting.

HttpContext.prototype.invoke seems to already support piping a stream provided by the remotable method to the HTTP response (source).

---

Tasks

• implementation in strong-remoting - see Stream as an accepts/returns argument strong-remoting#35 and Add support for "file" return args strong-remoting#284
• support in loopback-swagger - see Support the new return arg type "file" loopback-swagger#34
• documentation - see Doc: return response body from remote method #2209

A note for loopback-swagger support. Swagger spec says: http://swagger.io/specification/#responseObject

As an extension to the Schema Object, its root type value may also be "file". This SHOULD be accompanied by a relevant produces mime-type.

I don't remember whether loopback-swagger supports per-method produces option. If it does not, then we should implement it as part of this work.

Related: strongloop/loopback-component-explorer#103

[raymondfeng]

There are a couple of issues related to our strong-remoting stream support.

• It expects the stream is returned from the method call instead of the callback
• It only handles the stream, not headers such as Content-Type.

Maybe we should plug in an adapter into strong-remoting to handle multipart/form-data requests and have the storage-service upload function to be configured to the 'stream' binding.

[bajtos]

👍

This is my take on the API for stream responses (not a final version, just an idea):

function streaming(cb) {
  var bodyStream = // create response stream from file
  var contentType = // set from file extension
  cb(null, bodyStream, contentType);
}

loopback.remoteMethod(streaming, {
  returns: [
    { arg: 'bodyStream', type: 'stream', root: true },
    { arg: 'content-type', type: 'string', http: { target: 'header' } }
  ]
});

Not sure what API to use for request streams and multi-part requests.

[ritch]

+1

[bajtos]

Similar discussion: strongloop/strong-remoting#35

[haio]

Hi, any update on this issue? is there a example of how to implement download file functionality with loopback?
Thanks!

[bajtos]

@haio We don't have a proper support for streams implemented yet. The workaround of passing the HTTP response object to the remoted function should work though - see the links to loopback-storage-services in the issue description for more details.

File.download = function(file, res, cb) {
  var reader = fs.createReadStream('storage/' + file);
  reader.on('error', function (err) {
    res.type('application/json');
    res.send(500, { error: err });
  });
  reader.pipe(res);
  // callback is intentionally not invoked
};

File.remoteMethod('download', {
  accepts: [
    {arg: 'file', type: 'string', 'http': {source: 'path'}},
    {arg: 'res', type: 'object', 'http': {source: 'res'}}
  ],
  http: {
    verb: 'get',
    path: '/download/:file'
  }
});

[haio]

@bajtos I follow your way and see it works, thanks!

[tchambard]

Hi, I followed your server side implementation example to implement my own streaming from mongoDb gridfs storage, but I would be very interesting by the client side solution.
What I do not understand is how to pass the http response object as second argument...
I'm using Angular SDK and lbServices... but even with $http, I don't know how to do it...
Thanks for your help.

[tchambard]

I found the issue for my concern...
I needed to use res.send(500).send({ error: err }) instead of res.send(500, { error: err });

[benawarez]

Could you share how you implemented gridfs storage? Thanks.

[bajtos]

Closing in favour of follow-up issues: strongloop/loopback-swagger#34 and #2209

[web-hollywood]

Hi bajtos

I'm currently trying to download a file from the backend API(loopback3).
So I integrated it like this.

UserProfile.remoteMethod (
'totalDeductions',
{
description: 'Total deductions for exporting',
accepts: [
{arg: 'sDate', type: 'date', description: 'Start date for the report.', required: true},
{arg: 'eDate', type: 'date', description: 'End date for the report.', required: true},
{arg: 'res', type: 'object', 'http': {source: 'res'}}
],
isStatic: true,
returns: [
{ arg: 'body', type: 'file', root: true },
{ arg: 'Content-Type', type: 'string', http: { target: 'header' } },
{ arg: 'Content-Disposition', type: 'string', http: { target: 'header' } },
],
http: {verb: 'get', path: '/total-deductions'}
}
);

UserProfile.totalDeductions = function(sDate, eDate, res, cb) {
var DeductionModel = app.models.Deduction;
var filter = {
where: {
dateCreated: {between: [sDate,eDate]}
}
};

DeductionModel.find(filter, function(err, deductions) {
  var conf = {};
  conf.name = "mysheet";
  conf.cols = [{
    caption:'string',
        type:'string',
        beforeCellWrite:function(row, cellData){
       return cellData.toUpperCase();
    },
        width:28.7109375
  },{
    caption:'date',
    type:'date',
    beforeCellWrite:function(){
      var originDate = new Date(Date.UTC(1899,11,30));
      return function(row, cellData, eOpt){
              if (eOpt.rowNum%2){
                eOpt.styleIndex = 1;
              }  
              else{
                eOpt.styleIndex = 2;
              }
                if (cellData === null){
                  eOpt.cellType = 'string';
                  return 'N/A';
                } else
                  return (cellData - originDate) / (24 * 60 * 60 * 1000);
      } 
    }()
  },{
    caption:'bool',
    type:'bool'
  },{
    caption:'number',
     type:'number'        
  }];
  conf.rows = [
    ['pi', new Date(Date.UTC(2013, 4, 1)), true, 3.14],
    ["e", new Date(2012, 4, 1), false, 2.7182],
        ["M&M<>'", new Date(Date.UTC(2013, 6, 9)), false, 1.61803],
        ["null date", null, true, 1.414]  
    ];
  var result = nodeExcel.execute(conf);
  if (err) {
    cb(err);
    return;
  }
  var contentType =  'application/force-download';
  var contentDisposition = 'attachment; filename=xrt.pdf';
  cb(null, result,contentType, contentDisposition);
});

But I'm not able to download a file. It still returns response as a contect, not a file.
Can you please help me?

[bajtos]

@web-hollywood I don't see any obvious problem. Please create a small app that I can use to reproduce the issue on my computer (see http://loopback.io/doc/en/contrib/Reporting-issues.html#bug-report) and open a new issue. Thanks!