Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ Question ] How do you manipulate proxy response #97

Open
iroy2000 opened this issue Aug 6, 2016 · 32 comments
Open

[ Question ] How do you manipulate proxy response #97

iroy2000 opened this issue Aug 6, 2016 · 32 comments

Comments

@iroy2000
Copy link

@iroy2000 iroy2000 commented Aug 6, 2016

Hi, I'm using Express 4,

In normal way, I can do the following

    app.get('/v1/users/:username', function(request, response, next) {
        var username = request.params.username;
        findUserByUsername(username, function(error, user) {
            if (error) return next(error);
            return response.render('user', user);
        });
    });

But how do I execute custom logic if I'm using proxy, let's say I want to manipulate some of the data before response to the user? Is there a good pattern to do that with this middleware ?

app.use('/api', proxy({target: 'http://www.example.org', changeOrigin: true}));

@chimurai

This comment has been minimized.

Copy link
Owner

@chimurai chimurai commented Aug 6, 2016

I don't have the answer unfortunately.

You are not the first one the ask this question. (#91 (comment))

Happy to add this one to the http-proxy-middleware recipes.

Maybe this example might help you:
https://github.com/nodejitsu/node-http-proxy/blob/master/examples/middleware/modifyResponse-middleware.js

Can you post this question on StackOverflow (with backlink)? Hopefully your question will get an answer there.

Thanks!

@iroy2000

This comment has been minimized.

Copy link
Author

@iroy2000 iroy2000 commented Aug 9, 2016

I tried to use https://github.com/langjt/node-http-proxy-json

And it seems to solve my problem. Note that the library is still fairly new, but it's definitely take out some pain points from many people that has the same issues.

Let me just quote my code here for people who is interested, Hope it helps :)

@chimurai You can even add that to your recipe if you think it is good enough.

import express from 'express';
import proxy from 'http-proxy-middleware';
import config from 'config';
import modifyResponse from 'node-http-proxy-json';

const router = new express.Router();

const myProxy = proxy(config.get('some.json.api.Url'), {
  changeOrigin: true,
  logLevel: 'debug',
  pathRewrite: (path, req) => path.replace('/api', '/_ajax'),
  onProxyRes(proxyRes, req, res) {

    delete proxyRes.headers['content-length'];

    modifyResponse(res, proxyRes.headers['content-encoding'], function (body) {
      if (body) {
        // modify some information
        body.version = 2;
        body.props = {
          "nestedProps": true
        };
      }
      return body;
    });
  },
});


router.use('/api/example', myProxy);

export default router;

@silverprize

This comment has been minimized.

Copy link

@silverprize silverprize commented Sep 27, 2016

@iroy2000 Thanks for your approach. But I'd like to add custom header from body's property.
I tried and faced "Can't set headers after they are sent.".
I suggest to use multi proxy module.
express-http-proxy provide async proxy response.
express-http-proxy is inconvenient than http-proxy-middleware thus keep http-proxy-middleware as main proxy.
Few apis that need to modify header or body are called via express-http-proxy.

@cokeknight

This comment has been minimized.

Copy link

@cokeknight cokeknight commented Dec 20, 2016

here is my answer,

onProxyRes :function(proxyRes, req, res){
      var _write = res.write;
      var output;
      var body = "";
      proxyRes.on('data', function(data) {
          data = data.toString('utf-8');
          body += data;
      });
      res.write = function (data) {
        try{
          eval("output="+body)
          output = mock.mock(output)
          _write.call(res,JSON.stringify(output));
        } catch (err) {}
      }
    }

add onProxyRes option on the http-proxy-middleware
use the data event on the proxyRes to get the output
then modify the output in res.write

@enl31

This comment has been minimized.

Copy link

@enl31 enl31 commented Mar 10, 2017

@cokeknight thank you, can you help me more? How do you change header in this approach?
I want to have custom header containgn length of (json) body in response like this:

  res.write = data => {
    data = data.toString('utf-8');
    body += data;
    try{
      body = JSON.parse( body);
       res.set({
        'Content-Type': 'Application/json',
        'count': body.meta.count
      });
      _write.call(res,JSON.stringify(body.data));
    } catch (err) {}
  }
@zaaack

This comment has been minimized.

Copy link

@zaaack zaaack commented May 5, 2017

Not working for async case, I need return some async data, but it just return nothing.

@benjamin658

This comment has been minimized.

Copy link

@benjamin658 benjamin658 commented Jul 7, 2017

@enl31 You can set content-length by this way:

proxyRes.headers['content-length'] = data.length;

@westmark

This comment has been minimized.

Copy link

@westmark westmark commented Jul 12, 2017

This was my final solution (inspired by @kokarn 's solution over at http-party/node-http-proxy#796

I'm using this in a webpack dev server setup by create-react-app, hence the comment further down.

onProxyRes: ( proxyRes, req, res ) => {
    const _writeHead = res.writeHead;
    let _writeHeadArgs;
    const _end = res.end;
    let body = '';

    proxyRes.on( 'data', ( data ) => {
      data = data.toString( 'utf-8' );
      body += data;
    } );

    // Defer writeHead
    res.writeHead = ( ...writeHeadArgs ) => {
      _writeHeadArgs = writeHeadArgs;
    };

    // Defer all writes
    res.write = () => {};

    res.end = ( ...endArgs ) => {
      // Do everything in the end. Means we wont be streaming the response but who cares in a dev env..
      const output = body
        .replace( /stufftoreplace/g, '/leweb' );

      if ( proxyRes.headers && proxyRes.headers[ 'content-length' ] ) {
        res.setHeader( 'content-length', output.length );
      }

      // This disables chunked encoding
      res.setHeader( 'transfer-encoding', '' );

      // Disable cache for all http as well
      res.setHeader( 'cache-control', 'no-cache' );

      _writeHead.apply( res, _writeHeadArgs );

      if ( body.length ) {
         // Write everything via end()
        _end.apply( res, [ output ] );
      } else {
        _end.apply( res, endArgs );
      }
    };
  }
@youngzhaosignifyd

This comment has been minimized.

Copy link

@youngzhaosignifyd youngzhaosignifyd commented Aug 12, 2017

If proxyRes is gzipped, I will need to unzip it before pass it to res. Is there a better way to work around this?

Here is my code

onProxyRes: (proxyRes, req, res) => {
    const end = res.end;
    const writeHead = res.writeHead;
    let writeHeadArgs;
    let body; 
    let buffer = new Buffer('');

    // Concat and unzip proxy response
    proxyRes
      .on('data', (chunk) => {
        buffer = Buffer.concat([buffer, chunk]);
      })
      .on('end', () => {
        body = zlib.gunzipSync(buffer).toString('utf8');
      });

    // Defer write and writeHead
    res.write = () => {};
    res.writeHead = (...args) => { writeHeadArgs = args; };

    // Update user response at the end
    res.end = () => {
      const output = manipulateBody(body); // some function to manipulate body
      res.setHeader('content-length', output.length);
      res.setHeader('content-encoding', '');
      writeHead.apply(res, writeHeadArgs);

      end.apply(res, [output]);
    };
  }
@nickgrealy

This comment has been minimized.

Copy link

@nickgrealy nickgrealy commented Oct 15, 2017

I'm currently using @cokeknight 's solution (#97 (comment)), and I'm finding my JSON responses are being truncated after 8192 characters.

Is anyone else finding this?

N.B. It could be (probably is) because I'm not using the following code snippet... can someone explain what the intention is, and why you wouldn't just do _write.call(res, body);?

eval("output="+body)
output = mock.mock(output)
_write.call(res, JSON.stringify(output));

(e.g. assign body reference to variable output (why use eval?), then wrap it in a mock?)

@ortonomy

This comment has been minimized.

Copy link

@ortonomy ortonomy commented Feb 18, 2018

Allright, I also used : https://github.com/langjt/node-http-proxy-json

and it solved my problem. I recommend it over the manual re-writing of res.write methods - it the same thing as the solutions above, but it's captured in a library. 😅

@sesm

This comment has been minimized.

Copy link

@sesm sesm commented Mar 1, 2018

Perhaps the core problem is that response proxy gets from original source is not passed through existing express middleware chain?
Express already has an ecosystem of middleware, but for some reason we have to duplicate it's functionality in onProxyRes. This doesn't seem right.

@TitasGailius

This comment has been minimized.

Copy link

@TitasGailius TitasGailius commented Mar 7, 2018

I've been trying to set a cookie which is extracted from the response body but headers are already sent after the response is read.
Any ideas on how could I defer sending headers until I read the response?

This is one of the ways I tried with
https://github.com/langjt/node-http-proxy-json

{
    onProxyRes: (proxyRes, req, res) => {
        modifyResponse(res, proxyRes, body => {
            res.cookies.set('foo', 'bar')
        })
    }
}
@sergeymakoveev

This comment has been minimized.

Copy link

@sergeymakoveev sergeymakoveev commented Mar 14, 2018

@youngzhaosignifyd @westmark
Fix it

res.setHeader('content-length', output.length);

to

res.setHeader('content-length', Buffer.byteLength(output));
@garajo

This comment has been minimized.

Copy link

@garajo garajo commented May 11, 2018

I needed to res.redirect(). Thanks to the solutions above, here is the variance. Binding with res prevents undefined this errors.

onProxyRes: function (proxyRes, req, res) {
  const _writeHead = res.writeHead.bind(res);

  ...

  proxyRes.on('end', () => {
    const data = JSON.parse(body)
    res.writeHead(301, { Location: req.headers.referer })
    res.end()

  })
})
@sergeymakoveev

This comment has been minimized.

Copy link

@sergeymakoveev sergeymakoveev commented May 11, 2018

@garajo
try _writeHead(301, 'ok', { Location: req.headers.referer })
instead res.writeHead(301, { Location: req.headers.referer })

@sergeymakoveev

This comment has been minimized.

Copy link

@sergeymakoveev sergeymakoveev commented May 11, 2018

Guys, you can use my universal helper for a proxy and modify the response:
https://gist.github.com/sergeymakoveev/cc453586a59896c7605c22382d181bb7#file-webpack-response-modify__-babel-js

@garajo

This comment has been minimized.

Copy link

@garajo garajo commented May 11, 2018

@sergeymakoveev I think you meant this, res.writeHead(301, 'ok', { Location: req.headers.referer }), which worked. Thank you! p.s. I don't need to modify my response

@NoraGithub

This comment has been minimized.

Copy link

@NoraGithub NoraGithub commented Jun 10, 2018

Is there an official solution? Asking for help.

@NoraGithub

This comment has been minimized.

Copy link

@NoraGithub NoraGithub commented Jun 10, 2018

@sergeymakoveev Thanks for your code. I read it. If the framework doesn't call res.writeHead would result in some problems, like I use in my koa server not webpack_dev_server. It just use res.setHeader.

@Njerschow

This comment has been minimized.

Copy link

@Njerschow Njerschow commented Jun 20, 2018

I have tried the solutions above and unfortunately I am getting a junked body. Here is my code:

onProxyRes : function(proxyRes, req, res) {
            var body = "";
            proxyRes.on('data', function(data) {
                data = data.toString( 'utf-8' );
                body += data;
                console.log(body.toString('utf8'));
            });
},

and here is my output:

�iD����,�Y Ս����� &�GHUN뮬�2�d2V�%m�\�ݵu�m��erfĬ;u�
6���e��*�#a��w�L�}9C �9E"a2𑥔C\;�!�zc�Z�@l$<� ��t�)XA�r�QiL<B�Y�_���1Kze�$�f@�c�uJ�E��EJ�q��y).���c�/#�eۓm}�V3���y�t}�d��N-�p�)L�/��o�-*�_�媚�uEDȽ����)�-�8 V����>R���6�ζ��4\��M�M��Ǿ��W���LJ�3|�u�J�=Z���G��ƏP-��+��E'�KLT�ޘ������줠��k=�{��B��G�������A
7�3�r��<�)�'8��Z��>^IU5[;���F��8��Y�ܘM1���k9�nj�S�Y�jQ]����v��Ma�`

I am expecting a json response, but everything I've tried doesn't seem to be working.

Thank you for response in advance.

@Njerschow

This comment has been minimized.

Copy link

@Njerschow Njerschow commented Jun 20, 2018

I solved my own problem, the response from the server was gzipped, so I simply set the "accept-encoding" header like so:
'accept-encoding': 'gzip;q=0,deflate,sdch'

@xingshijie

This comment has been minimized.

Copy link

@xingshijie xingshijie commented Jun 29, 2018

The easy method

onProxyRes: function (proxyRes, req, res) {
    const _pipe = proxyRes.pipe
    // can get resp body from proxyRes pipe
    
    var body = new Buffer('test1234');

    proxyRes.pipe = function (res) {
      res.write(body)
      res.end()
    }
  }
@ViggoV

This comment has been minimized.

Copy link

@ViggoV ViggoV commented Sep 5, 2018

Can anyone tell me what type proxyRes is? Documentation features an example or two but I'm desperately looking for an overview/api reference of how it actually works..

@ViggoV

This comment has been minimized.

Copy link

@ViggoV ViggoV commented Sep 5, 2018

Just had a minor epiphany and double checked.. proxyRes is a Node stream.Readable instance.. Might be valuable to include in the docs :)

@mrt123

This comment has been minimized.

Copy link

@mrt123 mrt123 commented Sep 20, 2018

My solution:

const zlib = require('zlib');
...
onProxyRes: (proxyRes, req, res) => {
	let originalBody = new Buffer('');
	proxyRes.on('data', function (data) {
		originalBody = Buffer.concat([originalBody, data]);
	});
	proxyRes.on('end', function () {
		const bodyString = zlib.gunzipSync(originalBody).toString('utf8')
		const objectToModify = JSON.parse(bodyString)
		objectToModify.modification = 'Mickey Mouse'
		res.end(JSON.stringify(objectToModify));
	});
},
selfHandleResponse: true . // necessary to avoid res.end being called automatically
...
@eschaefer

This comment has been minimized.

Copy link

@eschaefer eschaefer commented Oct 30, 2018

@mrt123 's response got us 90% there. But if you need to gzip your response before sending it back out, here's how to do that:

 onProxyRes: (proxyRes, req, res) => {
  let originalBody = Buffer.from([]);
  proxyRes.on('data', data => {
    originalBody = Buffer.concat([originalBody, data]);
  });

  proxyRes.on('end', () => {
    const bodyString = zlib.gunzipSync(originalBody).toString('utf8');
    const newBody = doSomeReplacementStuff(bodyString);

    res.set({
      'content-type': 'text/html; charset=utf-8',
      'content-encoding': 'gzip'
    });
    res.write(zlib.gzipSync(newBody));
    res.end();
  });
},
selfHandleResponse: true,
@dvelasquez

This comment has been minimized.

Copy link

@dvelasquez dvelasquez commented Jan 22, 2019

I'm looking to do something similar, but transforming the encoding from latin1 to utf-8. I also need to preserve the original headers, but I'm getting unformatted data.

@sachinmour

This comment has been minimized.

Copy link

@sachinmour sachinmour commented Feb 28, 2019

I think this is the correct way to do it according to the official documentation modify-response

app.use('/api', proxy({
  target: 'http://www.example.org',
  changeOrigin: true,
  selfHandleResponse: true, // so that the onProxyRes takes care of sending the response
  onProxyRes: function(proxyRes, req, res) {
    var body = new Buffer('');
    proxyRes.on('data', function(data) {
      body = Buffer.concat([body, data]);
    });
    proxyRes.on('end', function() {
      body = body.toString();
      console.log("res from proxied server:", body);
      res.end("my response to cli");
    });
  }
}));
@devinrhode2

This comment has been minimized.

Copy link

@devinrhode2 devinrhode2 commented Mar 26, 2019

Not sure why, but when I use @sachinmour's code the "res from proxied server:" has encoding errors, and I mostly see diamonds with question marks inside. Furthermore, when I try to load a page, it gets downloaded, and viewing the file I can see the same encoding errors that I see in terminal or chrome console :(

@devinrhode2

This comment has been minimized.

Copy link

@devinrhode2 devinrhode2 commented Mar 26, 2019

Lo-and-behold, the previous example from @eschaefer does not have these encoding errors. I guess I needed gzip...!

@andyseonet

This comment has been minimized.

Copy link

@andyseonet andyseonet commented Nov 18, 2019

Sorry for adding to an old thread but this may save someone hours and potentially days in searching for a solution to removing headers from an http-proxy response.

The way of using selfHandleResponse seems to be a little bit of a blunt instrument for me and it didn't take into consideration streaming and forces the nodejs to buffer the full response before sending and for large files this could be a problem.

I found by looking at the library that it would be possible to remove headers for example by looking for the proxyResp events without specifying the selfHandleResponse option, the important thing to note was to ensure that the headers are in the correct order as browsers can be very picky about the ordering of the headers.

It is indeed possible to simply to iterate and copy the headers across by object name however this will loose the header order and browsers will start to half load pages or just fail for some content types.

Here's a simplified version of proxyRes event listener with the logic i have used - this will allow the proxy to stream as it should and remove headers for example if you don't want to disclose the apache or tomcat version or whatever header you wish to hide.

I'm not sure to the extent that it will handle any complex domain rewrites etc that you might have setup and you should test and handle them but it's worth it to keep streaming imho.

let headersToRemove = ['X-Forwarded-For','Apache-Version',........];

proxy.on('proxyRes', function (proxyRes, req, res) {
  if(!res.headersSent){
    proxyRes.headers = {};
    for(let x = 0; x < proxyRes.rawHeaders.length; x+=2) {
      if(headersToRemove .indexOf(proxyRes.rawHeaders[x]) == -1) { 
        proxyRes.headers[proxyRes.rawHeaders[x]] = proxyRes.rawHeaders[x+1];
      }
    }
  }
});

Hope it helps someone.

Cheers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.