Add truncation to request/response bodies

[billyvg]

Let's add truncation to request/resp bodies. For strings it should be pretty straightforward. For JSON we could consider a library like https://www.npmjs.com/package/truncate-json -- ideally this is done (and loaded) in the webworker.

[mydea]

So, after some investigation, these are the options we have for truncating request/response bodies:

1. No truncation at all

Just capture full bodies, full stop.
This is not feasible as we should guard against extreme payloads.

2. Send payloads up to a given size, then drop

This is basically the current experimental implementation: If the payload is <= a given size (e.g. 300kb), we capture it, else we drop it.

3. Truncate payloads based on max. size

We send the full payload, but if it is larger than a given max. size (e.g. 300kb) we truncate it to the given max size.

There are some variants for how to do this:

3.a) Truncate the string, even if JSON

We can just cut off the body at the given max size. This means if the body was JSON, it will afterwards just be a plain string. While this is very straightforward to do, it has two main downsides: 1. It can be hard to explore an incomplete JSON string, 2. PII scrubbing will have a harder time and will often just redact the whole payload if it finds any PII in it.

3.b) Smart-truncate JSON

This means truncating the JSON string, but ensuring the result is still valid JSON. This can be done by parsing the truncated string and ensuring any missing tokens are added back to the string. This means that you'll basically get the first x (e.g. 300k) characters of the JSON, afterwards the remainder of the JSON object will be removed/replaced by placeholders.

For reference, this could look like this:

{
  "user": [
    { "name": "~~" }
  ]
}

Where it truncated the name and everything after it.

We investigated a few different approaches and found one that allows us to do this with linear complexity, meaning it should be fine to do this on the main thread.

3.c Truncate based on depth

A variant of 3.b, instead of truncating after a given size is reached, we could truncate when a given depth is reached. This has the advantage that you may get a broader view of your response, but it may still be very large (as you could have large elements in the shallow entries of the JSON), and will also always hide deeper elements even for small but heavily nested payloads.

My 2 cents

After investigating all of these options, I believe we can go with option 3b) for the best user experience. I propose a max. size of 150kb. We will add meta information to the bodies that allow us to know:

• This body has been truncated as JSON
• This body has been truncated as a string (e.g. form data, ...)

[billyvg]

3b sounds great to me, I'd be curious how often users would need the full payload. To me, having 150KB of your response should be enough to debug issues.