Composeable content decoding & transformation tools

[pimterry]

Right now, HTTP Toolkit supports a fixed set of content formatting tools (here) and content decoding libraries (in https://www.npmjs.com/package/http-encoding), which are treated as separate concerns, only applied to content in the body, and can only be applied in quite limited ways (encoding applied automatically based on headers, one formatting selectable from the dropdown).

This is often limiting, e.g. JSON data encoded as base64 can be decoded from base64 to show the raw data, but can't then subsequently be auto-highlighted & formatted as JSON.

Some other tools such as Burp provide more flexible options to select arbitrary content in the UI, and run a series of transformations on it:

It would be interesting to explore an HTTP Toolkit native version of this. In effect: a way of defining a pipeline of transformations, and applying those. Doing so for the message body is still the most likely use case, but making this possible for arbitrary text also seems helpful.

Being able to do this in an explorative way (viewing the results as you build a pipeline, to explore unknown formats) is important.

UX is tricky. We don't have an easy place to include side panels like this, and it's very challenging from a space-in-the-UI point of view. Maybe we could do:

• A separate pipeline page, where you can build pipelines with lots of transformations, and manage a library of them
• An option to pick any existing pipeline as a format to show the message body
• A "Send to pipeline" button next to headers, and available when selecting & right-clicking any text, which jumps to the pipeline page with that data.

('Pipeline' term to be discussed later, lots of possible ways to describe this depending on how it turns out)

Would that work?

Might even be interesting to ship the same UI as a standalone app too - I can imagine lots of other non-HTTP cases where you might want to pass data through transformations like this, seems like it's just be a useful thing to make in general.

This is a substantial project, but quite an interesting one. Contributions welcome if anybody wants to take this on, and any other feedback on this idea below (particularly the UX) would be very helpful.

---

Does this affect you too? Click below and add a 👍 to vote for this and help decide where HTTP Toolkit goes next, or go vote on the other most popular ideas so far.

[dsalex1]

@pimterry CyberChef seems like a perfect fit for these sort of transformations you're looking for. As the project is open source it might be possible to integrate it directly, as some sort of embed, or linking from httptoolkit to a url in the web demo of cyberchef (the input as well as the pipeline steps can be passed in the query params)

otherwise the ui of cyberchef could be a great example of what a dedicated pipeline page might look like, i personally find the ui of cyberchef very intuitive, yet very powerful for all sort of transformation/decoding tasks.

[pimterry]

Thanks @dsalex1! That's very helpful, concrete examples like this are really useful. I'll definitely look through CyberChef's optins in detail for designing this, looks like a great tool and there's lots of useful features there we should aim to include here too.