hanging POST requests #180

Closed
joeyAghion opened this Issue Jan 25, 2012 · 31 comments

Comments

Projects
None yet

Hi,
I'm using express and trying to proxy certain requests to a different server, as follows (simplified):

var httpProxy = require('http-proxy');
var apiProxy = new httpProxy.RoutingProxy();
...
app.all('/api/*', function(req, res){
  apiProxy.proxyRequest(req, res, {host: 'example.com', port: 9000});
});

...but POST requests to my server appear to just hang, and never arrive at the target server.

I'm working on a stripped-down-but-working example of this but thought I'd ask first. Has anyone seen this behavior, or are there good examples of node-http-proxy coexisting with express?
Thanks!

Contributor

AvianFlu commented Jan 25, 2012

I believe your issue may be solved by using the buffer method. A very good explanation can be found here.

@AvianFlu: Thanks for the buffer pointer, but this ended up being caused by express's body-parsing.

Instead of declaring the API route (for proxying) within express routes, I configured it as middleware and used it before express.bodyParser().

Some coffeescript of the solution:

httpProxy = require 'http-proxy'
routingProxy = new httpProxy.RoutingProxy()
apiProxy = (pattern, host, port) ->
  (req, res, next) ->
    if req.url.match(pattern)
      routingProxy.proxyRequest req, res, host: host, port: port
    else
      next()

app.configure ->
  @set 'views', 'app/templates'
  # ...
  @use apiProxy /\/api\/.*/, 'localhost', 3000
  @use express.bodyParser()
  # ...
Contributor

AvianFlu commented Jan 26, 2012

You're correct - express's body parsing turns the request stream into a regular object. The solution is either to buffer it, as I was suggesting, or to deal with the request before the parsing, as you ended up doing. Glad you figured it out!

AvianFlu closed this Jan 26, 2012

@drewp drewp added a commit to drewp/node-http-proxy that referenced this issue Feb 7, 2012

@drewp drewp Address ticket #180 here since that problem is so hard to discover wh…
…en you run into it. If there was an error, people would search for the error text, but there isn't.
73e415a

@indexzero indexzero pushed a commit that referenced this issue Feb 12, 2012

Charlie Robbins Merge pull request #189 from drewp/master
proposed doc addition for #180
91bb2a9

I ran into this problem too .. http-proxy@0.8.0, express@2.5.8, connect@1.8.5.

In my case, it turns out that http-proxy's buffer doesn't play well with connect's bodyParser middleware. The buffer is adding listeners for the 'data' and 'end' events to the request object so that it can re-emit them later (on a call to proxyRequest, e.g.). The bodyParser middleware is also adding listeners for the 'data' and 'end' events to the request object but they are never removed. So when re-emitted by the buffer, they're picked up a second time by the bodyParser.

This can result in JSON parse errors and "Can't render headers after they are sent to the client" exceptions (and probably other errors).

eethann commented Dec 7, 2012

For those having issues with this, I found that this information seemed a bit out of date or difficult to implement (too much so for me, at least), compared to using connect-restreamer. That approach is outlined in the middleware example for using the bodyParser: https://github.com/nodejitsu/node-http-proxy/blob/master/examples/middleware/bodyDecoder-middleware.js

ijse commented Dec 30, 2012

So, How to solve this problem?

connect-restreamer worked for me. The last thing in my middle-ware is:

app.use(require('connect-restreamer')());

Then I have a can do the proxy request normally:

var httpProxy = require('http-proxy');
var routingProxy = new httpProxy.RoutingProxy();

app.get('/proxy/', function (req, res) {
  routingProxy.proxyRequest(req, res, {
    host : host,
    port : port
  });
});

Configuring the proxy as part of middleware before bodyParser (if this is an option for you) also worked for me.

I did not attempt the buffer solution.

xixixao commented May 23, 2013

Also, simply not using the bodyParser also works (for anyone's future reference).

This thread has helped me. Thanks everyone.

Helpful, Thanks!

For some reason, connect-restreamer didn't fix it for me.

If anyone wants yet another fix, I ended up just calling bodyParser manually. So the proxying worked since bodyParser is not in the middleware. The code looked like:

var connect = require('connect');
var bodyParser = connect.bodyParser();

...

app.post("/blah", function(req, res) {
    bodyParser(req, res, function() {
        // Now do something with req.body.thePostData
        ...
        res.send(200);
    });
});

@imrehorvath imrehorvath added a commit to tombenke/restapi that referenced this issue Dec 12, 2013

@imrehorvath imrehorvath Fix issue with the order of middlewares
The order of middlewared does matter. Using the bodyParser before the
httpProxy does break the requests with JSON body.

For more detailed discussion refer to
nodejitsu/node-http-proxy#180
387a852

fonini commented Feb 6, 2014

@andyfischer

Man, you saved my life!

Thanks for that solution it works fine
Just configure the proxy as part of middleware before bodyParser

gdw2 commented Nov 6, 2014

Restreamer works for me. I needed to leave bodyParser because my proxy was conditional upon the contents of the request body. Here is a working example authored by someone else:

https://github.com/nodejitsu/node-http-proxy/blob/master/examples/middleware/bodyDecoder-middleware.js

MightyPixel referenced this issue in balderdashy/sails Nov 17, 2014

Closed

Explicit body parser #2410

FoxxMD commented Apr 14, 2015

The example from bodyDecoder pointed out by @gdw2 does not work for me. I get

/Code/project/node_modules/http-proxy/lib/http-proxy/index.js:119
    throw err;
          ^
Error: write after end
    at ClientRequest.OutgoingMessage.write (_http_outgoing.js:413:15)
    at IncomingMessage.ondata (_stream_readable.js:540:20)
    at IncomingMessage.emit (events.js:107:17)
    at /Code/project/lib/server.js:46:25
    at process._tickCallback (node.js:355:11)

Using connect-restreamer results in the request continuing to hang.

I am using express instead of connect.

var express = require('express'),
    bodyParser = require('body-parser'),
    httpProxy = require('http-proxy');

var proxy = httpProxy.createProxyServer({changeOrigin: true});
var restreamer = function (){
  return function (req, res, next) { //restreame
    req.removeAllListeners('data')
    req.removeAllListeners('end')
    next()
    process.nextTick(function () {
      if(req.body) {
        req.emit('data', req.body)
      }
      req.emit('end')
    })
  }
}

var app = express();

app.use(bodyParser.urlencoded({extended: false, type: 'application/x-www-form-urlencoded'}));
app.use(restreamer());

app.all("/api/*", function(req, res) {
    //modifying req.body here
    //
    proxy.web(req, res, { target: 'http://urlToServer'});
});

app.listen(8080);

This may related to this issue however.

@FoxxMD same behavior here

For those in the unfortunate position not being able to rearrange/remove the bodyparser middleware:
I've added the snippet from the example bodyDecode middleware right before I use proxy.web(...) and it just werkz

[...]
req.removeAllListeners('data');
req.removeAllListeners('end');

process.nextTick(function () {
if(req.body) {
   req.emit('data', JSON.stringify(req.body));
}
req.emit('end');
});
proxyServer.web( [...] );
[...]

Hope that helps. This was really frustrating.

glebtv commented Apr 30, 2015

restreamer didn't work for me.
Only way I was able to get it working with routing is like this:

app.post '/api/*', (req, res)->
  r = new http.IncomingMessage()
  r.httpVersion = req.httpVersion
  r.method = req.method
  r.headers = req.headers
  delete r.headers['accept-encoding']
  r.url = req.url
  r.socket = req.socket
  body = JSON.stringify(req.body)
  r.body = body
  size = Buffer.byteLength(body, 'utf8')
  r.headers['content-type'] = "application/json;charset=UTF-8"
  r.headers['content-length'] = size
  buffer = {}
  buffer.pipe = (dest)->
    process.nextTick ->
      dest.write(body)
  proxy.web(r, res, {
    buffer: buffer
    target: "http://localhost:1600"
    proxyTimeout: 5000
  })

@schumacher-m thanks for that snippet...works for me

@schumacher-m BIG thanks for your tips. It works around the hang issue finally - after a whole day of debugging.

dmkc commented Nov 3, 2015

Thanks @schumacher-m . There's actually also an example in the repo itself that handles the same issue: https://github.com/nodejitsu/node-http-proxy/blob/master/examples/middleware/bodyDecoder-middleware.js

Thanks all - I went with OP's suggestion and moved the proxy route above the bodyParser middleware

vvpvvp commented Mar 2, 2016

I used all of the solution above, _but it's not worked anyway_.
So, I read the http-proxy source and request API, I found that's the question of request.
and http-proxy have no solution with the body parameters .
There is no need to use a middleware.

And this is my solution:

var proxy = httpProxy.createProxyServer();
proxy.on('proxyReq', function(proxyReq, req, res, options) {
    if(req.method=="POST"&&req.body){
        proxyReq.write(req.body);
        proxyReq.end();
    }
});

var headers = {};
if(req.method=="POST"&&req.body){
    var data = JSON.stringify(req.body);
    req.body = data;
    headers = { 
        "Content-Type": "application/json",
        "Content-Length": data.length
    }
}
proxy.web(req, res, {
    target: url,
    headers:headers
});

phi4grv referenced this issue in parti-xyz/auth-ui Mar 21, 2016

Closed

Modify post data in http-proxy #30

@vvpvvp,Can i modify/add req.newkey = 'a' and send it to the proxy

narciero commented Apr 29, 2016 edited

None of these solutions work with the latest version of express & http-proxy, any suggestions? Here is what my code looks like - I still get the hanging POST request issue. I am just trying to read the POST body before conditionally proxying the request.

var httpProxy = require('http-proxy'),
    proxy = httpProxy.createProxyServer({ ignorePath: true });

var express = require('express');
var app = express();

app.use(bodyParser.json());
app.use(restreamer());

app.post('/', function(req, res) {
 proxy.web(req, res, { target: URL });
}

EDIT: I am now getting a 'write after end' error which is odd because I am using restreamer.

gutenye commented Jun 20, 2016

@vvpvvp big thinks for your tips

@narciero I am also getting same error. Did you find any workaround for this ?

Putting this before using the express body parser worked for me! Thanks for the help guys.

rajiff commented Nov 5, 2016

Hi

I have done slightly similar yet different fix for this, without using any restreamer, take a look here

Idea is to not use the body-parser as middleware instead apply only to the matched routes in my local web server and hence don't bring in the body-parser at all in between proxy and the local web server

Comments, suggestions welcome

breeswish referenced this issue in redfin/react-server Mar 17, 2017

Closed

Take control all server middlewares? #900

differui commented Jun 15, 2017 edited

Here is my solution:

import { createServer as createHttpServer } from 'http'
import { createProxyServer } from 'node-http-proxy'

function parseBody(message) {
  const body = []

  return new Promise((resolve, reject) => {
    message.on('data', c => body.push(c))
    message.on('end', () => resolve(body))
    message.on('error', err => reject(err))
  })
}

export default function createServer(proxyCfg, port) {
  const proxyServer = createProxyServer(proxyCfg)
    .on('proxyReq', onProxyReq)

  function onProxyReq(proxyReq, req, res) {
    if (req.body && req.body.length) {
      proxyReq.write(new Buffer(req.body[0]))
      proxyReq.end()
    }
  }
  createHttpServer(async (req, res) => {
    req.body = parseBody(req)
    if (!res.headersSent) {
      proxyServer.web(req, res)
    }
  }).listen(port)
}

vshih commented Jun 23, 2017

The bodyDecoder-middleware.js example shows a similar implementation of this:

//restream parsed body before proxying
proxy.on('proxyReq', function(proxyReq, req, res, options) {
  if(req.body) {
    let bodyData = JSON.stringify(req.body);
    // incase if content-type is application/x-www-form-urlencoded -> we need to change to application/json
    proxyReq.setHeader('Content-Type','application/json');
    proxyReq.setHeader('Content-Length', Buffer.byteLength(bodyData));
    // stream the content
    proxyReq.write(bodyData);
  }
});

Amazing, this issue is 5 years old...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment