How to modify the response HTML data going back to the user? #382

PhilAndrew · 2013-03-13T04:21:33Z

Hi, I really need to know how to get the response HTML as a string and modify it.

This question has been asked before at stackoverflow but I don't see how to do it.

http://stackoverflow.com/questions/13596942/using-node-js-to-proxy-http-and-modify-response

In general, I need to get access to all request data and all response data and change the response data in some cases. Thanks!

indexzero · 2013-03-13T05:34:44Z

This short answer is you can't. You can observe it, but you cannot modify it, node-http-proxy writes to the response for you. In 0.9.0 we add the proxyResponse event which allows you to observe the response data for caching purposes, but not to modify it.

e.g.

//
// Create a proxy server with custom application logic
//
httpProxy.createServer(function (req, res, proxy) {
  //
  // Listen to the `proxyResponse` event. Make note that 
  // the first two arguments to this handler are the `req`
  // and `res` in the existing scope and I'm indicating that 
  // they are meaningless by assigning them the var name `_` 
  //
  req.on('proxyResponse', function (_, _, proxyRes) {
    proxyRes.on('data', function (chunk) {
      //
      // This is the data from the target server, but modifying
      // it will not affect the outgoing `res`. 
      // 
    });
  });

  //
  // Put other custom server logic here
  //

  // Now make the proxy request.
  proxy.proxyRequest(req, res, {
    host: 'localhost',
    port: 9000
  });
}).listen(8000);

In general this is really bad practice unless you can do the modification in a streaming manner because you will have to buffer the entire response.

PhilAndrew · 2013-03-14T05:42:08Z

Hi there,

What I want to do is have a Wordpress website and proxy it with NodeJS, then when a webpage comes back from Wordpress at some particular URL then I want to modify the contents of the webpage to insert a form.

The idea is that its slightly difficult to develop in Wordpress and easier to develop in NodeJS (for me). So by proxy Wordpress I thought I can add extra things to my Wordpress page. Wordpress is good for business cases, fast development of site at a cheaper price and easy for user to use as a CRM.

Can you suggest a way to go about doing this? I would really appreciate any thoughts you have on this.

I want to intercept all incoming and outgoing and modify the request/response data in insert my own data, ie a form.

Thanks! Philip

No9 · 2013-03-14T09:29:24Z

@PhilHongKong Harmon is designed to plug into node-http-proxy https://github.com/No9/harmon
It uses trumpet and so is stream based to work around the buffering problem that @indexzero mentions. It uses an element and attribute selector to enable manipulation of a response.

indexzero · 2013-03-14T10:18:41Z

Nice. @No9 could you make a pull-request to README.md about this? We get this question a lot.

PhilAndrew · 2013-03-14T13:01:59Z

Thanks, yes my first thought was to google for a proxy to NodeJS. The question is in general, how to find the Harmon/Trumpet in the first place for a person who is searching for a way to change the data coming out of a web-server passing through the proxy, I didn't find Haron/Trumpet from google.

No9 · 2013-03-16T12:26:48Z

Pull done. Also did a little bit of house keeping on the repo readme too.

englishbulldog · 2013-12-04T12:25:04Z

How about JSON responses? What what module similar to this can I use to modify JSON responses?

Skyross · 2015-02-26T13:37:12Z

How about using replacestream?

akshayl · 2015-04-11T22:29:31Z

Harmon / Trumpet allow you to replace a html element, but what if you want to insert elements at various points in the document being returned?

No9 · 2015-04-11T22:49:35Z

Hi @akshayl
https://github.com/No9/harmon/blob/master/examples/rotate.js
Demonstrates adding an additional style tag in the head without replacing.

Does this fit your use case?

akshayl · 2015-04-11T23:52:42Z

Thanks for your reply @No9
That example seems to replace the entire head tag including the script tag which outputs the message "The piece of javascript also inside the head tag wasn't touched :)"

Before:
<html><head><script>window.onload = function () {document.getElementById("message").innerHTML = "The piece of javascript also inside the head tag wasn't touched :)";}</script></head><body><h3>A simple example of injecting some css to rotate an image into a page before it is rendered.</h3><image src="http://i.imgur.com/fpMGL.png" /><div id="message"></div></body></html>

After:
<html><head><style type="text/css"> img { -webkit-transform: rotate(-90deg); -moz-transform: rotate(-90deg); filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=3);}</style></head><body><h3>A simple example of injecting some css to rotate an image into a page before it is rendered.</h3><image src="http://i.imgur.com/fpMGL.png" /><div id="message"></div></body></html>

No9 · 2015-04-12T00:13:15Z

@akshayl ah now I get you.
So https://github.com/No9/harmon/blob/master/examples/doge.js#L22
Shows how you can hold the content of the node and then use it afterwards.
The sample just logs it to console but I think this might be what you are looking for?

shlomihassan · 2015-04-30T11:40:33Z

hi guys i am working with the same solution

//
// Create a proxy server with custom application logic
//
httpProxy.createServer(function (req, res, proxy) {
//
// Listen to the proxyResponse event. Make note that
// the first two arguments to this handler are the req
// and res in the existing scope and I'm indicating that
// they are meaningless by assigning them the var name _
//
req.on('proxyResponse', function (_, _, proxyRes) {
proxyRes.on('data', function (chunk) {
//
// This is the data from the target server, but modifying
// it will not affect the outgoing res.
//
});
});

//
// Put other custom server logic here
//

// Now make the proxy request.
proxy.proxyRequest(req, res, {
host: 'localhost',
port: 9000
});
}).listen(8000);

but in sum cases i am getting only half of the data
is there a way to get all the data no mater the size?

shlomi

ndarilek · 2016-04-26T14:49:31Z

Sorry to hijack this issue, but I want to do exactly what this issue says I should be able to. I want to inject an httpProxy between clients and a JSON API server, only observe the JSON data before passing it back, and perform server-side logic. This seems like it should be easy but it isn't. Here is my code:

proxy.on("proxyRes", (proxyReq, req, res) => {
  res.on("data", (msg) => {
    console.log("msg", msg)
  })
  res.on("end", () => {
    console.log("end")
  })
  res.on("finish", (data) => {
    console.log("finish", data)
  })
})

I just get a bunch of finish events and no data. Using this issue because it was what I found when I searched, and I'd have expected the code above to work but either I'm putting it in the wrong place or this solution is outdated.

I'm stuck on Node 0.10.43 if that makes a difference--not my choice, unfortunately.

ndarilek · 2016-04-26T15:00:25Z

And of course 5 minutes after I post that issue, I spot my typo:

proxy.on("proxyRes", (proxyRes, req, res) => {

I.e. The first parameter to the event callback is a res, not a req. Sorry for the spam folks. The lesson for the day is coffee first, then code.

Armalon · 2016-08-10T13:56:25Z

I had same problem and only solution I've found I peeped in the very same harmon module: https://github.com/No9/harmon/blob/master/examples/gzipped.js#L47-L64
When the author creates an server which listens 9000 port he uses such a hack when he gzipes the response. He uses zlib.Gzip which is a Transform stream and forwards his regular data through it to gzip it.
So we could do the same, here is the example:

var http = require('http'),
    connect = require('connect'),
    httpProxy = require('http-proxy');
var replaceStream = require('replacestream');
var app = connect();
var proxy = httpProxy.createProxyServer({});

app.use(function(req, res) {
    // Magic starts HERE
    // Put here any Transform stream you wish, I used replaceStream, 
    // you can write your own using following method 
    // http://codewinds.com/blog/2013-08-20-nodejs-transform-streams.html#creating-transform-stream-which-uppercases-all-text
    var replace = replaceStream('script', 'NOSCRIPT');
    var _write = res.write;
    var _end = res.end;

    replace.on('data', function(buf){
        _write.call(res, buf);
    });
    replace.on('end', function(){
        _end.call(res);
    });

    res.write = function(data){
        replace.write(data);
    };
    res.end = function(){
        replace.end();
    };
    // Magic ends HERE

    proxy.web(req, res, { target: 'http://' + req.headers.host });
});

var proxyServer = http.createServer(app);
console.log("listening on port 5050")
proxyServer.listen(5050);

Or you can just use https://github.com/philippotto/transformer-proxy

albanm · 2017-03-02T22:01:48Z

I needed to apply some throttling to node-http-proxy, so not really a transformation of the body but still the best implementation that I could think of is to use a throttle stream (for example node-stream-throttle). As I didn't want a hackish solution, I made a small addition to the options in this fork: koumoul-dev@4db8736

The resTransformStream and reqTransformStream options are exactly what their names suggest: optional transform streams that can be piped between req to proxyReq and proxyRes to res.

If some people are interested and the maintainers are ok with the idea I am willing to work on a proper PR, with doc, test, etc.

yorch · 2017-06-29T20:50:29Z

Found this package: resp-modifier that works fairly well for this usecase

trusktr · 2023-12-05T06:25:35Z

Is there still no easy built-in way to send a different response with http-proxy?

Use case: the proxy may only know when to replace a response with a 302 redirect based on looking at the content, but with the proxyRes event it is already too late to replace the response.

StanDarius24 · 2024-03-22T18:39:19Z

Hi @trusktr did you find a solution for this particular case?
I also want to modify the body when I receive a certain response code

Thanks!

indexzero closed this as completed Mar 13, 2013

No9 mentioned this issue Mar 16, 2013

Mention Harmon used for response modifications in the readme #384

Merged

kltm mentioned this issue Oct 8, 2014

Implement live synced view all the time geneontology/noctua#16

Closed

steve-jansen mentioned this issue Oct 26, 2016

enable CORS steve-jansen/json-proxy#30

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to modify the response HTML data going back to the user? #382

How to modify the response HTML data going back to the user? #382

PhilAndrew commented Mar 13, 2013

indexzero commented Mar 13, 2013

PhilAndrew commented Mar 14, 2013

No9 commented Mar 14, 2013

indexzero commented Mar 14, 2013

PhilAndrew commented Mar 14, 2013

No9 commented Mar 16, 2013

englishbulldog commented Dec 4, 2013

Skyross commented Feb 26, 2015

akshayl commented Apr 11, 2015

No9 commented Apr 11, 2015

akshayl commented Apr 11, 2015

No9 commented Apr 12, 2015

shlomihassan commented Apr 30, 2015

ndarilek commented Apr 26, 2016

ndarilek commented Apr 26, 2016

Armalon commented Aug 10, 2016 •

edited

Loading

albanm commented Mar 2, 2017

yorch commented Jun 29, 2017

trusktr commented Dec 5, 2023

StanDarius24 commented Mar 22, 2024

How to modify the response HTML data going back to the user? #382

How to modify the response HTML data going back to the user? #382

Comments

PhilAndrew commented Mar 13, 2013

indexzero commented Mar 13, 2013

PhilAndrew commented Mar 14, 2013

No9 commented Mar 14, 2013

indexzero commented Mar 14, 2013

PhilAndrew commented Mar 14, 2013

No9 commented Mar 16, 2013

englishbulldog commented Dec 4, 2013

Skyross commented Feb 26, 2015

akshayl commented Apr 11, 2015

No9 commented Apr 11, 2015

akshayl commented Apr 11, 2015

No9 commented Apr 12, 2015

shlomihassan commented Apr 30, 2015

ndarilek commented Apr 26, 2016

ndarilek commented Apr 26, 2016

Armalon commented Aug 10, 2016 • edited Loading

albanm commented Mar 2, 2017

yorch commented Jun 29, 2017

trusktr commented Dec 5, 2023

StanDarius24 commented Mar 22, 2024

Armalon commented Aug 10, 2016 •

edited

Loading