Allow option to disable the automatic body parsing in Gatsby Functions

[graysonhicks]

Some APIs require the original request to verify its authenticity. Currently it is parsed by Gatsby Functions so the request fail. Stripe webhooks are a good example. Gatsby should provide an API for config-per-function, with this being one of the options (e.g. bodyParsing: false).

[byronlanehill]

Have we seen any updates on this? I am currently trying to solve this problem and am unable to find any solutions.

[graysonhicks]

This is caused by the express.json middleware here. A short term solution would be to add a rawBody field to the req explicitly by adding a middleware function like this:

function addRawBody(req, res, next) {
  req.setEncoding('utf8');

  var data = '';

  req.on('data', function(chunk) {
    data += chunk;
  });

  req.on('end', function() {
    req.rawBody = data;

    next();
  });
}

A more flexible and powerful solution would be reading a config export off the function. The default export is already read and executed here. Any other export could be found the same way. A possible solution is to remove the automatic express.json() call and making a bodyParser custom middleware function that tests for a rawBody option like this:

function addRawBody(req, res, next) {
    // Get function  
    const { "0": pathFragment } = req.params

    const { functions }: { functions: Array<IGatsbyFunction> } =
      store.getState()

    // Get function object.
    let functionObj = functions.find(
      ({ functionRoute }) => functionRoute === pathFragment
    )
  
  // Check for config.  If not using rawBody option, just use express middleware.
  if(!functionObj.config.rawBody){
      express.json(req, res, next)
  } else {
    // Otherwise set the body as raw
    req.setEncoding('utf8');
      var data = '';

    req.on('data', function(chunk) {
      data += chunk;
    });

    req.on('end', function() {
      req.body = data;
 
      next();
    });
  }
}

Realistically, adding a config object would involve more planning and documentation, but is a long term better solution.

[KyleAMathews]

@byronlanehill tried this and it doesn't work as the request is consumed by the body-parser before this middleware is hit so it can no longer access the raw body. See the edit on this stackoverflow answer https://stackoverflow.com/a/18710277

[graysonhicks]

Ah, okay, looks like the verify option could do it:

fiznool/body-parser-xml#2 (comment)

var express = require('express'),

var app = express();
app.use(express.json({
  verify: function(req, res, buf, encoding) {
    if(buf && buf.length) {
    // Get function  
    const { "0": pathFragment } = req.params
    const { functions }: { functions: Array<IGatsbyFunction> } =
      store.getState()

    // Get function object.
    let functionObj = functions.find(
      ({ functionRoute }) => functionRoute === pathFragment
    )
      
   // Check for config.  If using rawBody option, add rawBody field.
    if(functionObj.config.rawBody){
       req.rawBody = buf.toString(encoding || 'utf8');
    } 
    }
  }
}));

[KyleAMathews]

yes — this works but we still don't want to do this as then user would have two copies of the request body in memory which would be a big footgun.

Most people want the parsed body. If they want the escape hatch, we should give them direct access to the low-level API. Only high or low level APIs work. In-between ones just create messes.

[KyleAMathews]

You solve for the 80-90% of use cases and give people a low-level API so they can solve all the other niche use cases. There's many reasons you might want to have direct access to the request stream and getting access to the raw body bytes is just one of them.

[graysonhicks]

Yea, makes sense. So what would the low-level override API look like?

[KyleAMathews]

Something like:

export const config = {
  bodyParsing: false
}

[ehowey]

Just popping in to say that I just spent a couple hours trying to figure out why my code wasn't running with the Stripe webhooks API - and found this thread eventually. I wasn't expecting Gatsby to be parsing the incoming data - but also makes sense when I think about it. +1 for creating a config option to disable this.

[graysonhicks]

Ah! I saw you were working on that on Twitter and thought about pointing this out (Stripe is one of the few that really needs this) but assumed you already saw. My bad!

[ehowey]

No worries Grayson! Hopefully this can be implemented sometime later this year after the GatsbyConf pressure winds down.

Missed your reply somehow!

[njbmartin]

I've just run into issues creating webhooks for Discord. Like Stripe, it also requires validating the signature, and needs access to the rawBody. You can't register a webhook without it :(

[raae]

Yeah for shipping this, however I cannot seem to get it to work. I followed the docs and this is my code is my code: queen-raae/gatsby-funcjam-21#4. Working in develop. I am testing this using gatsby develop.

[raae]

Is this supposed to work locally using develop @graysonhicks ? I still get the body as json, and stripe construct event complains.

[graysonhicks]

I think @EzraEllette worked on this one, any ideas? @raae First comment says working in develop, but second one implies it doesn't, can you clarify?

[raae]

I meant I am doing this in develop as in "I am working in develop". Sorry about the confusion. Tested today by pushing to Cloud and there I do get the raw body. I have not tested build + serve yet.

[graysonhicks]

Okay, thanks! And sorry it looks like @pieh did the function work, and Ezra did the docs.

PR is here: https://github.com/gatsbyjs/gatsby/pull/35393/files

I definitely see serve changes: https://github.com/gatsbyjs/gatsby/pull/35393/files#diff-a332f30c333a3b353bd79e3cfcf2d552cfefd9ed79a78ec36803654a093cdb5c

[Khaledgarbaya]

This is fixed now in #35780

[Khaledgarbaya]

This is released now in gatsby 4.14