Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hanging POST requests #180

Closed
joeyAghion opened this issue Jan 25, 2012 · 35 comments

Comments

Projects
None yet
@joeyAghion
Copy link

commented Jan 25, 2012

Hi,
I'm using express and trying to proxy certain requests to a different server, as follows (simplified):

var httpProxy = require('http-proxy');
var apiProxy = new httpProxy.RoutingProxy();
...
app.all('/api/*', function(req, res){
  apiProxy.proxyRequest(req, res, {host: 'example.com', port: 9000});
});

...but POST requests to my server appear to just hang, and never arrive at the target server.

I'm working on a stripped-down-but-working example of this but thought I'd ask first. Has anyone seen this behavior, or are there good examples of node-http-proxy coexisting with express?
Thanks!

@AvianFlu

This comment has been minimized.

Copy link
Contributor

commented Jan 25, 2012

I believe your issue may be solved by using the buffer method. A very good explanation can be found here.

@joeyAghion

This comment has been minimized.

Copy link
Author

commented Jan 26, 2012

@AvianFlu: Thanks for the buffer pointer, but this ended up being caused by express's body-parsing.

Instead of declaring the API route (for proxying) within express routes, I configured it as middleware and used it before express.bodyParser().

Some coffeescript of the solution:

httpProxy = require 'http-proxy'
routingProxy = new httpProxy.RoutingProxy()
apiProxy = (pattern, host, port) ->
  (req, res, next) ->
    if req.url.match(pattern)
      routingProxy.proxyRequest req, res, host: host, port: port
    else
      next()

app.configure ->
  @set 'views', 'app/templates'
  # ...
  @use apiProxy /\/api\/.*/, 'localhost', 3000
  @use express.bodyParser()
  # ...
@AvianFlu

This comment has been minimized.

Copy link
Contributor

commented Jan 26, 2012

You're correct - express's body parsing turns the request stream into a regular object. The solution is either to buffer it, as I was suggesting, or to deal with the request before the parsing, as you ended up doing. Glad you figured it out!

@AvianFlu AvianFlu closed this Jan 26, 2012

drewp added a commit to drewp/node-http-proxy that referenced this issue Feb 7, 2012

Address ticket nodejitsu#180 here since that problem is so hard to di…
…scover when you run into it. If there was an error, people would search for the error text, but there isn't.

indexzero pushed a commit that referenced this issue Feb 12, 2012

Charlie Robbins
Merge pull request #189 from drewp/master
proposed doc addition for #180
@coldfire22x

This comment has been minimized.

Copy link

commented Feb 16, 2012

I ran into this problem too .. http-proxy@0.8.0, express@2.5.8, connect@1.8.5.

In my case, it turns out that http-proxy's buffer doesn't play well with connect's bodyParser middleware. The buffer is adding listeners for the 'data' and 'end' events to the request object so that it can re-emit them later (on a call to proxyRequest, e.g.). The bodyParser middleware is also adding listeners for the 'data' and 'end' events to the request object but they are never removed. So when re-emitted by the buffer, they're picked up a second time by the bodyParser.

This can result in JSON parse errors and "Can't render headers after they are sent to the client" exceptions (and probably other errors).

@eethann

This comment has been minimized.

Copy link

commented Dec 7, 2012

For those having issues with this, I found that this information seemed a bit out of date or difficult to implement (too much so for me, at least), compared to using connect-restreamer. That approach is outlined in the middleware example for using the bodyParser: https://github.com/nodejitsu/node-http-proxy/blob/master/examples/middleware/bodyDecoder-middleware.js

@ijse

This comment has been minimized.

Copy link

commented Dec 30, 2012

So, How to solve this problem?

@cesutherland

This comment has been minimized.

Copy link

commented Jan 14, 2013

connect-restreamer worked for me. The last thing in my middle-ware is:

app.use(require('connect-restreamer')());

Then I have a can do the proxy request normally:

var httpProxy = require('http-proxy');
var routingProxy = new httpProxy.RoutingProxy();

app.get('/proxy/', function (req, res) {
  routingProxy.proxyRequest(req, res, {
    host : host,
    port : port
  });
});

Configuring the proxy as part of middleware before bodyParser (if this is an option for you) also worked for me.

I did not attempt the buffer solution.

@xixixao

This comment has been minimized.

Copy link

commented May 23, 2013

Also, simply not using the bodyParser also works (for anyone's future reference).

@renatoargh

This comment has been minimized.

Copy link

commented Jun 27, 2013

This thread has helped me. Thanks everyone.

@nicolazj

This comment has been minimized.

Copy link

commented Jul 19, 2013

Helpful, Thanks!

@andyfischer

This comment has been minimized.

Copy link

commented Sep 13, 2013

For some reason, connect-restreamer didn't fix it for me.

If anyone wants yet another fix, I ended up just calling bodyParser manually. So the proxying worked since bodyParser is not in the middleware. The code looked like:

var connect = require('connect');
var bodyParser = connect.bodyParser();

...

app.post("/blah", function(req, res) {
    bodyParser(req, res, function() {
        // Now do something with req.body.thePostData
        ...
        res.send(200);
    });
});

imrehorvath added a commit to tombenke/restapi that referenced this issue Dec 12, 2013

Fix issue with the order of middlewares
The order of middlewared does matter. Using the bodyParser before the
httpProxy does break the requests with JSON body.

For more detailed discussion refer to
nodejitsu/node-http-proxy#180
@fonini

This comment has been minimized.

Copy link

commented Feb 6, 2014

@andyfischer

Man, you saved my life!

@tarek-salah

This comment has been minimized.

Copy link

commented Oct 29, 2014

Thanks for that solution it works fine
Just configure the proxy as part of middleware before bodyParser

@gdw2

This comment has been minimized.

Copy link

commented Nov 6, 2014

Restreamer works for me. I needed to leave bodyParser because my proxy was conditional upon the contents of the request body. Here is a working example authored by someone else:

https://github.com/nodejitsu/node-http-proxy/blob/master/examples/middleware/bodyDecoder-middleware.js

@FoxxMD

This comment has been minimized.

Copy link

commented Apr 14, 2015

The example from bodyDecoder pointed out by @gdw2 does not work for me. I get

/Code/project/node_modules/http-proxy/lib/http-proxy/index.js:119
    throw err;
          ^
Error: write after end
    at ClientRequest.OutgoingMessage.write (_http_outgoing.js:413:15)
    at IncomingMessage.ondata (_stream_readable.js:540:20)
    at IncomingMessage.emit (events.js:107:17)
    at /Code/project/lib/server.js:46:25
    at process._tickCallback (node.js:355:11)

Using connect-restreamer results in the request continuing to hang.

I am using express instead of connect.

var express = require('express'),
    bodyParser = require('body-parser'),
    httpProxy = require('http-proxy');

var proxy = httpProxy.createProxyServer({changeOrigin: true});
var restreamer = function (){
  return function (req, res, next) { //restreame
    req.removeAllListeners('data')
    req.removeAllListeners('end')
    next()
    process.nextTick(function () {
      if(req.body) {
        req.emit('data', req.body)
      }
      req.emit('end')
    })
  }
}

var app = express();

app.use(bodyParser.urlencoded({extended: false, type: 'application/x-www-form-urlencoded'}));
app.use(restreamer());

app.all("/api/*", function(req, res) {
    //modifying req.body here
    //
    proxy.web(req, res, { target: 'http://urlToServer'});
});

app.listen(8080);

This may related to this issue however.

@franck34

This comment has been minimized.

Copy link

commented Apr 20, 2015

@FoxxMD same behavior here

@schumacher-m

This comment has been minimized.

Copy link

commented Apr 28, 2015

For those in the unfortunate position not being able to rearrange/remove the bodyparser middleware:
I've added the snippet from the example bodyDecode middleware right before I use proxy.web(...) and it just werkz

[...]
req.removeAllListeners('data');
req.removeAllListeners('end');

process.nextTick(function () {
if(req.body) {
   req.emit('data', JSON.stringify(req.body));
}
req.emit('end');
});
proxyServer.web( [...] );
[...]

Hope that helps. This was really frustrating.

@aidanbon

This comment has been minimized.

Copy link

commented Jul 28, 2015

@schumacher-m BIG thanks for your tips. It works around the hang issue finally - after a whole day of debugging.

@dmkc

This comment has been minimized.

Copy link

commented Nov 3, 2015

Thanks @schumacher-m . There's actually also an example in the repo itself that handles the same issue: https://github.com/nodejitsu/node-http-proxy/blob/master/examples/middleware/bodyDecoder-middleware.js

@homerjam

This comment has been minimized.

Copy link

commented Feb 22, 2016

Thanks all - I went with OP's suggestion and moved the proxy route above the bodyParser middleware

@vvpvvp

This comment has been minimized.

Copy link

commented Mar 2, 2016

I used all of the solution above, _but it's not worked anyway_.
So, I read the http-proxy source and request API, I found that's the question of request.
and http-proxy have no solution with the body parameters .
There is no need to use a middleware.

And this is my solution:

var proxy = httpProxy.createProxyServer();
proxy.on('proxyReq', function(proxyReq, req, res, options) {
    if(req.method=="POST"&&req.body){
        proxyReq.write(req.body);
        proxyReq.end();
    }
});

var headers = {};
if(req.method=="POST"&&req.body){
    var data = JSON.stringify(req.body);
    req.body = data;
    headers = { 
        "Content-Type": "application/json",
        "Content-Length": data.length
    }
}
proxy.web(req, res, {
    target: url,
    headers:headers
});
@tomalex0

This comment has been minimized.

Copy link

commented Apr 19, 2016

@vvpvvp,Can i modify/add req.newkey = 'a' and send it to the proxy

@narciero

This comment has been minimized.

Copy link

commented Apr 29, 2016

None of these solutions work with the latest version of express & http-proxy, any suggestions? Here is what my code looks like - I still get the hanging POST request issue. I am just trying to read the POST body before conditionally proxying the request.

var httpProxy = require('http-proxy'),
    proxy = httpProxy.createProxyServer({ ignorePath: true });

var express = require('express');
var app = express();

app.use(bodyParser.json());
app.use(restreamer());

app.post('/', function(req, res) {
 proxy.web(req, res, { target: URL });
}

EDIT: I am now getting a 'write after end' error which is odd because I am using restreamer.

@gutenye

This comment has been minimized.

Copy link

commented Jun 20, 2016

@vvpvvp big thinks for your tips

@akprats33

This comment has been minimized.

Copy link

commented Jul 3, 2016

@narciero I am also getting same error. Did you find any workaround for this ?

@dansantner

This comment has been minimized.

Copy link

commented Sep 3, 2016

Putting this before using the express body parser worked for me! Thanks for the help guys.

@rajiff

This comment has been minimized.

Copy link

commented Nov 5, 2016

Hi

I have done slightly similar yet different fix for this, without using any restreamer, take a look here

Idea is to not use the body-parser as middleware instead apply only to the matched routes in my local web server and hence don't bring in the body-parser at all in between proxy and the local web server

Comments, suggestions welcome

@differui

This comment has been minimized.

Copy link

commented Jun 15, 2017

Here is my solution:

import { createServer as createHttpServer } from 'http'
import { createProxyServer } from 'node-http-proxy'

function parseBody(message) {
  const body = []

  return new Promise((resolve, reject) => {
    message.on('data', c => body.push(c))
    message.on('end', () => resolve(body))
    message.on('error', err => reject(err))
  })
}

export default function createServer(proxyCfg, port) {
  const proxyServer = createProxyServer(proxyCfg)
    .on('proxyReq', onProxyReq)

  function onProxyReq(proxyReq, req, res) {
    if (req.body && req.body.length) {
      proxyReq.write(new Buffer(req.body[0]))
      proxyReq.end()
    }
  }
  createHttpServer(async (req, res) => {
    req.body = parseBody(req)
    if (!res.headersSent) {
      proxyServer.web(req, res)
    }
  }).listen(port)
}
@vshih

This comment has been minimized.

Copy link

commented Jun 23, 2017

The bodyDecoder-middleware.js example shows a similar implementation of this:

//restream parsed body before proxying
proxy.on('proxyReq', function(proxyReq, req, res, options) {
  if(req.body) {
    let bodyData = JSON.stringify(req.body);
    // incase if content-type is application/x-www-form-urlencoded -> we need to change to application/json
    proxyReq.setHeader('Content-Type','application/json');
    proxyReq.setHeader('Content-Length', Buffer.byteLength(bodyData));
    // stream the content
    proxyReq.write(bodyData);
  }
});

Amazing, this issue is 5 years old...

ndarilek added a commit to loconomics/loconomics-backend that referenced this issue Feb 9, 2018

@kevteljeur

This comment has been minimized.

Copy link

commented Jul 18, 2018

For anyone who has this issue of POST requests timing out through the proxy because of the post body, the middleware solution linked above (https://github.com/nodejitsu/node-http-proxy/blob/master/examples/middleware/bodyDecoder-middleware.js) worked for me. However, it did have a 'gotcha' - the code looks for an exact match on the contentType header, which might not happen:

	proxy.on( 'proxyReq', ( proxyReq, req, res, options ) => {
		if ( !req.body || !Object.keys( req.body ).length ) {
			return;
		}

		let contentType = proxyReq.getHeader( 'Content-Type' );
		let bodyData;

		if ( contentType.includes( 'application/json' ) ) {
			bodyData = JSON.stringify( req.body );
		}

		if ( contentType.includes( 'application/x-www-form-urlencoded' ) ) {
			bodyData = queryString.stringify( req.body );
		}

		if ( bodyData ) {
			proxyReq.setHeader( 'Content-Length', Buffer.byteLength( bodyData ) );
			proxyReq.write( bodyData );
		}
	});

Hope this helps someone who comes here (I should really put it on StackOverflow, shouldn't I?).

@Mtnt

This comment has been minimized.

Copy link

commented Aug 3, 2018

@kevteljeur

This comment has been minimized.

Copy link

commented Aug 4, 2018

As per the linked example for the full code: https://nodejs.org/dist/latest-v8.x/docs/api/querystring.html

@loretoparisi

This comment has been minimized.

Copy link

commented Sep 26, 2018

@vshih @kevteljeur thank you, I have tried this modified approach, that now is also in the examples, but for some reason it does not forward the body as json and I get an error. Asked on SF as well.

The error by middleware side was

[Tue Sep 25 2018 18:16:29 GMT+0200 (CEST)] trapped error code: message:Can't set headers after they are sent. stack:Error: Can't set headers after they are sent.
    at validateHeader (_http_outgoing.js:491:11)
    at ClientRequest.setHeader (_http_outgoing.js:498:3)
    at ProxyServer.<anonymous> (/webservice/lib/api.js:616:34)
    at ProxyServer.emit (/webservice/node_modules/eventemitter3/index.js:210:27)
    at ClientRequest.<anonymous> (/webservice/node_modules/http-proxy/lib/http-proxy/passes/web-incoming.js:132:27)
    at emitOne (events.js:121:20)
    at ClientRequest.emit (events.js:211:7)
    at tickOnSocket (_http_client.js:652:7)
    at onSocketNT (_http_client.js:668:5)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9):

The error on the backend (tornado) was Malformed HTTP message from 172.17.0.1: Malformed HTTP headers: '\r\n0\r\n\r\n'

while I did the reastream like

//restream parsed body before proxying
                proxy.on('proxyReq', function (proxyReq, req, res, options) {
                    if (!req.body || !Object.keys(req.body).length) {
                        return;
                    }
                    var contentType = proxyReq.getHeader('Content-Type');
                    var bodyData;

                    if (contentType === 'application/json') {
                        bodyData = JSON.stringify(req.body);
                    }

                    if (contentType === 'application/x-www-form-urlencoded') {
                        bodyData = queryString.stringify(req.body);
                    }

                    if (bodyData) {
                        console.log("------", bodyData);
                        proxyReq.setHeader('Content-Length', Buffer.byteLength(bodyData));
                        proxyReq.write(bodyData);
                    }
                });
                proxy.on('proxyRes', function (proxyRes, req, res) {
                    modifyResponse(res, proxyRes, function (response) {
                        if (!MXMUtil.empty(response) && !MXMUtil.empty(response.message) && !MXMUtil.empty(response.message.body)) { // valid response
                            if (!MXMUtil.empty(response.message.body.error)) { // error response
                                var message = self.processor.createErrorMessage(500, '', response.message.body.error);
                                response = message;
                            } else { // valid response
                                var message = self.processor.prepareResponse(req_start, response.message.body);
                                response = message;
                            }
                        }
                        return response; // return value can be a promise
                    });
                });
                proxy.on('error', function (error, req, res) {
                    var errorMessage = self.processor.createErrorMessage(500, '', error.message);
                    res.send(errorMessage);
                    res.end();
                });
                proxy.web(req, res, {
                    //  url string to be parsed with the url module
                    target: target_uri,
                    //  true/false, Default: false - changes the origin of the host header to the target URL
                    changeOrigin: true,
                    //  true/false, Default: false - specify whether you want to ignore the proxy path of the incoming request 
                    ignorePath: true
                });

the express uses bodyParser like

        // express app
        this.app = express();
        // development || production
        this.app.set('env', env);
        // custom logging: only error codes
        this.app.use(morgan('combined', {
            skip: function (req, res) {
                return res.statusCode < 400;
            }
        }));
        // LP: handle Error: request entity too large
        this.app.use(bodyParser.json({limit: '50mb'}));
        this.app.use(bodyParser.urlencoded({limit: '50mb', extended: true}));
        this.app.use(cookieParser());
        this.app.use(compression()); // gzip compression

I have also tried a simpler approach that was:

if (req.body && req.body.length) {
                        proxyReq.setHeader('Content-Type','application/json');
                        proxyReq.write(new Buffer(req.body[0]));
                        proxyReq.end();
                    }

In that case I get a json decoding error, while the Can't set headers after they are sent disappeared. I think that this was due because in the examples above the proxyReq.end(); after proxyReq.write it is missing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.