Raw data?

[internalfx]

Is there any easy way to get to the raw request body. Or is adding such a feature a possibility?

[internalfx]

Wow, you guys work fast. Unfortunately I wasn't clear enough....

I'm integrating with Shopify webhooks specifically.

They send a POST request with a JSON body. bodyparser works great for this.

The problem is that I have to validate the HMAC, and to do so I have to get the raw_body of the response as well. So, I need the parsed body and the raw body.

One for verification and the other for data.

[fengmk2]

Get the ctx.req request stream and pipe to a HMAC reader.

@internalfx Please give us a full example codes?

[internalfx]

My process is further down the middleware pipeline.
Not all requests need to be validated.

Be patient with me, my understanding of the internals of koa, bodyparser and co-body is not that great.

Here was my cheap hack on line 38 of co-body/lib/json.js. This is working well.

  return raw(inflate(req), opts)
    .then(function (str) {
      try {
        return {
          body: parse(str),
          raw: str
        }
      } catch (err) {
        err.status = 400
        err.body = str
        throw err
      }
    })

I also modified bodyparser on line 71

  function * parseBody (ctx) {
    if ((detectJSON && detectJSON(ctx)) || ctx.request.is(jsonTypes)) {
      let result = yield parse.json(ctx, jsonOpts)
      if (result.body) {
        ctx.request.body = result.body
        ctx.request.raw_body = result.raw
      } else {
        ctx.request.body = result
      }
    } else if (ctx.request.is(formTypes)) {
      ctx.request.body = yield parse.form(ctx, formOpts)
    } else {
      ctx.request.body = {}
    }
  }
}

Now in my route policy, something very similar to a sailsjs policy. I can do this....

let crypto = require('crypto')

let checkSignature = function (data, hmac) {
  let secret = config.shopify.shared_secret

  let digest = crypto
    .createHmac('SHA256', secret)
    .update(data)
    .digest('base64')

  return (digest === hmac)
}

module.exports = function *() {
  let raw_body = this.request.raw_body
  let hmac = this.headers['x-shopify-hmac-sha256']

  if (checkSignature(raw_body, hmac)) {
    return true
  }

  this.throw(401, 'Shopify HMAC check failed.')
}

[dead-horse]

@internalfx this module is made for generic usage. you can try https://github.com/stream-utils/raw-body to get the body content and parse by yourself. :)

[internalfx]

I understand if this is not where you want to take the library.

The modifications I've made have solved the problem.

I didn't want to give up the automatic JSON detection and parsing. I also wanted the form parsing that is built in.

I may just release a fork and track changes to bodyparser, while providing an option for raw data to myself and others.

Thanks guys.

[libook]

I need to keep raw data too.
For verifying signature usage.
Would you please provide plugin functions or just ctx.rawBody?

[surfingtomchen]

@internalfx

the code in body parser should be changed to

function* parseBody(ctx) {
    if (enableJson && ((detectJSON && detectJSON(ctx)) || ctx.request.is(jsonTypes))) {
	
      let result = yield parse.json(ctx, jsonOpts);

      if (result.body) {

        ctx.request.raw_body = result.raw
        return result.body

      } else {

        return result
      }
    }

    if (enableForm && ctx.request.is(formTypes)) {
      return yield parse.form(ctx, formOpts);
    }
    if (enableText && ctx.request.is(textTypes)) {
      return yield parse.text(ctx, textOpts) || '';
    }
    return {};
  }
  
};

otherwise you will lose the ctx.request.body information

[chentsulin]

I have seen there is a verify option in express/bodyparser, and it makes this hack possible:

app.use(bodyparser.json({
  verify: function(req, res, buf) { 
    req.rawBody = buf;
  },
}));

@fengmk2 @dead-horse Should we support something like that? Or just avoid this middleware and rewrite the body parsing logic based on raw-body by myself?

[dead-horse]

I'd like to add a options to let co-body return the raw body.

[dead-horse]

then we can access the request's raw body with ctx.request.rawBody and do what you like by your self.

[chentsulin]

Sounds great. How could we name this option? rawBody: true? It will be cool if we can access raw body (ctx.request.rawBody) and parsed body (ctx.request.body) at the same time.

[chentsulin]

the read function in expressjs/body-parser: https://github.com/expressjs/body-parser/blob/master/lib/read.js#L38-L132

[d1y]

then we can access the request's raw body with ctx.request.rawBody and do what you like by your self.

请问 rawBody 现在有了吗? 我们遇到一个奇怪的问题, 或者说是我姿势不对

--data '{"ipv4":"192.168.1.1","note":"开发","ftp_user":"d1y","ftp_pwd":"middle","ftp_path":""}'

理论上, 我应该是要得到一个对象的

但实际上我在 ctx.request.body 获取的是

body:  { '{"ipv4":"192': { '168': [ [Object] ] } }

. 符号被自动转义了, 我能想到的解决办法就是拿到原始数据 rawBody, 然后自己用 qs 转成 object, 我试过转成base64 不过 base64 中有 =, 也被自动转义了, 请问老哥, 我这个怎么玩

[libook]

@d1y Please use English that your message can be valuable to other people all over the world.

https://github.com/koajs/bodyparser/blob/master/index.js#L78
You can get raw body with ctx.request.rawBody.

https://github.com/koajs/bodyparser/blob/master/index.js#L91
https://github.com/cojs/co-body/blob/master/index.js#L4
https://github.com/cojs/co-body/blob/master/lib/json.js#L49-L58
You can see that it is just JSON.parse

Welcome to Node.js v13.2.0.
Type ".help" for more information.
> JSON.parse('{"ipv4":"192.168.1.1","note":"开发","ftp_user":"d1y","ftp_pwd":"middle","ftp_path":""}')
{
  ipv4: '192.168.1.1',
  note: '开发',
  ftp_user: 'd1y',
  ftp_pwd: 'middle',
  ftp_path: ''
}

So the problem is not about the bodyparser.

Seems like a kind of serialization problem on your client ,or your upstream middleware modified your ctx.request.body .

Please check your code again, or provide a runnable demo for reproducing and debugging the problem.

[libook]

@d1y I have got the key.
The "Content-Type" header form your client is "application/x-www-form-urlencoded".
That your server treated the body as form-data. The request header should be "application/json".