Proxyman converts escaped characters into real ones and breaks JSON correctness

[ACATTAN]

Proxyman version? (Ex. Proxyman 1.4.3)

3.8.0

macOS Version? (Ex. mac 10.14)

12.5

Steps to reproduce

When sending a valid JSON which contains an escaped special character (e.g. \u000b), Proxyman converts it to its real value (0x0B) and forwards it this way to the server. This creates an invalid JSON at the server side.

If for example you send this request to the server (directly, without going through the proxy):

curl --request POST 'http://172.25.16.71:9094/v1/events' \
--header 'Content-Type: application/json' \
--data-raw '{
  "report": {
      "text":"hello\u000bworld!"
  }
}'

then the server gets it as-is, with the exact 6 characters - backslash, u, 0, 0, 0, b in the text field.

However, if you send the exact same request via Proxyman:

curl -x 'http://127.0.0.1:9090' --request POST 'http://172.25.16.71:9094/v1/events' \
--header 'Content-Type: application/json' \
--data-raw '{
  "report": {
      "text":"hello\u000bworld!"
  }
}'

then the \u000b is converted to 0x0B, which is wrong.

Expected behavior

Proxyman should forward the JSON request as-is without modifying \u000b into it real character, because if it does so - the validity of the JSON is broken.

This bug is btw rather important to us as it prevents some messages from being correctly parsed by our servers. Thanks 😄

[NghiaTranUIT]

Thanks. It's an interesting issue that I'm not aware of when implementing the cURL.

Since it blocks your current workflow, I will prioritize this bug ticket over the filter ticket, which you open yesterday 👍

Fixing it and send you a Beta build soon 💯

[ACATTAN]

Thanks so much!
Just to clarify - I sent you a cURL example just to clarify the issue, but the real traffic was from an iPhone, through Proxyman, to our servers. The iPhone sent "\u000b" as a string inside a JSON body, and Proxyman converted the \u000b string into the real ascii character 0x0B (vertical tab), and that's the issue. It should have left the \u000b string untouched.
Thanks again!

[NghiaTranUIT]

I tried to reproduce the bug, but no luck 🤔

Log Server

• Github: https://github.com/NghiaTranUIT/simple-server-testing
• Code:

const express = require("express");
const app = express();
const port = 3000;
// app.use(express.json()) 

app.use(
    express.json({
      limit: '5mb',
      verify: (req, res, buf) => {
        req.rawBody = buf.toString();
      },
    })
  );

app.get("/", (req, res) => {
  res.send("Hello World!");
});

app.post("/hexdata", (req, res) => {
    console.log('--- Hex data is called!');

    const body = req.body;
    console.log('Body = ', body);
    console.log('Raw Body = ', req.rawBody);
    res.json(body);
});

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`);
});

=> I use rawBody which pares the request body without changing any data.

Test case 1: (No Proxy mode) Send a cURL with an escape string in the body

• Confirm that the request body is intact on the Server Log

Test case 2: (Proxy to Proxyman) Send a cURL with an escape string in the body

=> Not sure, but the body is still intact. I don't see that Proxyman has modified the body.

Test case 2: (Proxy to Proxyman) Use Insomnia to send a request with an escape string in the body

=> Result is the same. The body still displays "data": "hello\u000bworld!"

[NghiaTranUIT]

Just wondering: Do you use any debugging tools? Map Local, Breakpoint, Scripting 🤔 The request body can be modified by these tools.

If you don't mind, what server you're using? Is it ExpressJS, Python, Go, or something. I would like to use the same server to reproduce it.

[ACATTAN]

Hey @NghiaTranUIT thanks a lot for looking into it!
I understand now when it happens. It happens only if the scripting engine of Proxyman is activated. I.e. if there's no script catching the request - then it goes through as-is. However, it's enough to have a script that does nothing - just implements the onRequest and onResponse methods - the issue occurs. I guess that when there's a script - Proxyman actually decodes the message and re-encodes it, and that's when the issue occurs.

I used this script, targeted at any URL:

When this script is active - the issue happens.

(regarding your question about the server - I used our own Java-based server so I can't send that one to you, but I think it doesn't matter. I think that once you activate a script as I just described you'll be able to reproduce the issue).
Thanks again!

[NghiaTranUIT]

Thanks for the hint. I'm able to reproduce the bug 🙌 it's from the Scripting.

[NghiaTranUIT]

If you don't mind, @ACATTAN please use this beta build: https://proxyman.s3.us-east-2.amazonaws.com/beta/Proxyman_3.8.0_Fix_Unicode_encoding_in_scripting.dmg

It works fine on my test cases. Please let me know the result 👍

[ACATTAN]

Sure, thanks! I will try it tomorrow and let you know.

[ACATTAN]

Hey @NghiaTranUIT I tried the version and it looks fine.
I noticed that when the character \u000b is passed through the proxy, it is converted to \\v, but this is OK, as \\v is just a different representation of the same character (vertical tab). When trying \u000c - it remained \u000c. So that's fine.

Btw, on a different (and a bit related note) - when JSON requests go through scripting, the order of the JSON gets changed randomly. I.e. if the request contains for example {"a":1, "b":2}, it may reach the server as {"b":2, "a":1}. I understand it's probably because you read it into a Dictionary where order doesn't matter. For the server's software it normally doesn't matter too. But for a person going over requests - instead of seeing them in a consistent order, the fields inside the JSON are randomly mixed up, which is a bit confusing. Charles for example doesn't change the order of the original JSON, since its re-writes are based on string manipulations (regex) and not on parsing into a dictionary. Maybe it's complex/impossible to change this behavior, but anyway I wanted to mention that 🙏

[NghiaTranUIT]

I noticed that when the character \u000b is passed through the proxy, it is converted to \v, but this is OK, as \v is just a different representation of the same character (vertical tab). When trying \u000c - it remained \u000c. So that's fine.

Thanks for letting me know 🎉

I understand it's probably because you read it into a Dictionary where order doesn't matter.

It's a known issue. The dictionary on Swift doesn't maintain the order. As a result, the key-order is messed up when it serializes into Data. This issue only happens when using the Scripting.

We don't use Regex like Charles since it's easier for users when manipulating the JS Object.

If the request/response is not modified by the Scripting, the order of the JSON Body is intact.