Middleware for custom request rewrite logic

[buger]

Middleware

Middleware is a program that accepts request and response payload at STDIN and emits modified requests at STDOUT. You can implement any custom logic like stripping private data, advanced rewriting, support for oAuth and etc.

                   Original request      +--------------+
+-------------+----------STDIN---------->+              |
|  Gor input  |                          |  Middleware  |
+-------------+----------STDIN---------->+              |
                   Original response     +------+---+---+
                                                |   ^
+-------------+    Modified request             v   |
| Gor output  +<---------STDOUT-----------------+   |
+-----+-------+                                     |
      |                                             |
      |            Replayed response                |
      +------------------STDIN----------------->----+

Middleware can be written in any language, see examples/middleware folder for examples.
Middleware program should accept the fact that all communication with Gor is asynchronous, there is no guarantee that original request and response messages will come one after each other. Your app should take care of the state if logic depends on original or replayed response, see examples/middleware/token_modifier.go as example.

Simple bash echo middleware (returns same request) will look like this:

while read line; do
  echo $line
end

Middleware can be enabled using --middleware option, by specifying path to executable file:

gor --input-raw :80 --middleware "/opt/middleware_executable" --output-http "http://staging.server"

Communication protocol

All messages should be hex encoded, new line character specifieds the end of the message, eg. new message per line.

Decoded payload consist of 2 parts: header and HTTP payload, separated by new line character.

Example request payload:

1 932079936fa4306fc308d67588178d17d823647c 1439818823587396305
GET /a HTTP/1.1
Host: 127.0.0.1

Example response payload:

2 8e091765ae902fef8a2b7d9dd960e9d52222bd8c 2782013
HTTP/1.1 200 OK
Date: Mon, 17 Aug 2015 13:40:23 GMT
Content-Length: 0
Content-Type: text/plain; charset=utf-8

Header contains request meta information separated by spaces. First value is payload type, possible values: 1 - request, 2 - original response, 3 - replayed response.
Next goes request id: unique among all requests (sha1 of time and Ack), but remain same for original and replayed response, so you can create associations between request and responses. Third argument varies depending on payload type: for request - start time, for responses - round-trip time.

HTTP payload is unmodified HTTP requests/responses intercepted from network. You can read more about request format here, here and here. You can operate with payload as you want, add headers, change path, and etc. Basically you just editing a string, just ensure that it is RCF compliant.

At the end modified (or untouched) request should be emitted back to STDOUT, keeping original header, and hex-encoded. If you want to filter request, just not send it. Emitting responses back is optional, and does not affect anything at the moment.

Advanced example

Imagine that you have auth system that randomly generate access tokens, which used later for accessing secure content. Since there is no pre-defined token value, naive approach without middleware (or if middleware use only request payloads) will fail, because replayed server have own tokens, not synced with origin. To fix this, our middleware should take in account responses of replayed and origin server, store originalToken -> replayedToken aliases and rewrite all requests using this token to use replayed alias. See examples/middleware/token_modifier.go and middleware_test.go#TestTokenMiddleware as example of described scheme.

Latest precompiled binaries:
https://www.dropbox.com/s/27chmbsqxrolvz4/gor_0.9.9_middleware_x64.tar.gz?dl=1
https://www.dropbox.com/s/vmeg5sexcleoo2e/gor_0.9.9_middleware_x86.tar.gz?dl=1

[buger]

Thanks to #160 we how have access to raw response output.

[houndci-bot]

Final newline missing.

[alexjurkiewicz]

Hi, this work looks really interesting! I currently rewrite JSON data by dumping to a file and editing certain fields with sed :D. This looks more robust, hope it lands soon.

[buger]

@alexjurkiewicz moreover it allows you to read both original and replayed response, and allow to implement complex stuff like oAuth (rewriting tokens), and much more.

[lovewhll]

Hi buger,

My middleware is written in java.
It can accept request at STDIN well.

Thanks

[buger]

@lovewhll ensure that requests emitted to STDOUT properly hex encoded, if you will still have issues after that, sharing your middleware code may help.

[lovewhll]

Hi buger,

The middleware is just a echo program.

the code:

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

import org.apache.commons.io.IOUtils;

public class Echo {

public static void main(String[] args) {
    if(args != null){
        for(String arg : args){
            System.out.println(arg);
        }

    }

    BufferedReader stdin = new BufferedReader(new InputStreamReader(
            System.in));
    String line = null;

    try {
        while ((line = stdin.readLine()) != null) {

            System.out.println(line);

        }
    } catch (IOException e) {
        IOUtils.closeQuietly(stdin);
    }
}

}

[lovewhll]

Hi buger,

I`d never seen 2 - original response. Is this a problem?

Thanks

[buger]

@lovewhll can you also show how you run gor with middleware (which inputs/outputs do you use), maybe issue happens with specific combinations of plugins.

[lovewhll]

Hi buger,

/usr/local/project/gor/gor -verbose=true -debug=true --output-http-stats=true -stats=true --input-file /usr/local/project/gor/requests.gor --middleware "/usr/local/project/gor/gor-echo.sh" --output-http "http://192.168.0.10:8090"

gor-echo.sh
#!/bin/sh
/usr/java/jdk1.7.0_72/bin/java -jar /usr/local/project/gor/gor-echo.jar

Thanks

[buger]

@lovewhll since new file format is human readable, can you ensure that responses included into "requests.gor" file?

[buger]

(and how you generated requests.gor file)

[lovewhll]

Hi buger,

/usr/local/project/gor/gor --input-raw :8090 --output-file /usr/local/project/gor/requests.gor

Thanks

[lovewhll]

Hi buger,

I use examples/middleware/echo_modifier.sh to run gor with middleware,it works well.
so the java middleware is the point.

What is the default charcter code?

Thanks

[buger]

@lovewhll should be utf

[arothian]

This looks great! The token_modifier example is pretty much the use case I have for this. Hope to try it out soon.

[buger]

@lovewhll i just tested your middleware and it worked great (i included it into examples folder). I included java into my Docker development image, and you can test it using make drun-2, which basically does: gor --input-file="./fixtures/requests.gor" --output-dummy=0 --verbose --debug --middleware "java -cp ./examples/middleware echo"

[lovewhll]

@buger it works well, thank you!