Feature request: Multiple output types

[tlwr]

Summary

Creating complex recipes often ends up being much more complicated than it has to be because we lack expressive output types.

Example: files

Zipping and unzipping files produces HTML output, which is very useful as an end state but not good when it is in the middle of a recipe. What I usually end up doing is strip HTML and then a find replace.

Example: PGP encryption

For both testing internally, and composing operations in the UI, having structured output here would be really useful.

Proposed solution

Compound types

Type: File
Attributes:
- String Name
- Date Modified
- Bytes Contents

Operation: Unzip
Input: Bytes
Output: List<File>

Operation: Display files
Input: List<File>
Output: HTML

Operation: Generate PGP Key pair
Input: Ignored
Output: Map<String, String>

Example output:

{
  "Public Key": "-----BEGIN...",
  "Private Key": "-----BEGIN..."
}

Operations could have an intrinsic toString or showOperation property in the case that the operation is the last operation in the recipe.

An example recipe to hash first file in a zip file

Unzip
Get element from list ( index = 0) // first file
Get contents from file
SHA2

An example with registers and a new unregister operation

Operation: Unregister
Input: Ignored
Output: String
Arguments:
- Output format (Text)

Outputs the contents of registers according to the format

Unzip
Get element from list ( index = 0) // first file
Get contents from file
Register ( Extractor = "(.*)" ) // contents to $R0
Generate PGP Key
Extract value from Map ( key = "Private key")
Register ( Extractor = "(.*)" ) // contents to $R1
Unregister ( Format = "$R0" ) // file from above now the input to next operatioin
PGP Sign ( Private key = "$R1" ) // sign unzipped file using private key
Register ( Extractor = "(.*)" ) // signed file now in $R2
Unregister ( Format = 
"
Signed file 
$R2

File
$R0

Private key
$R1
"
)

This is still a little bit half-baked in my head, and I would love your thoughts and feedback.

[n1474335]

Yeah, a very good idea.

For the next big refactoring, I'd like to completely redesign how operations are defined and this could definitely play into it.

Wouldn't it be nice if each operation looked something like this:

class FromBase64 extends Operation {
    constructor(input, args) {
        super()
        // handle args
    }

    run() {
        ...
    }

    toString() {
        ...
    }

    ...
}

It would then be possible to either run an entire recipe, or expose each class separately, allowing other people to use the operations in CyberChef in their projects:

import { FromBase64 } from "cyberchef";

FromBase64.run("SGVsbG8sIFdvcmxkIQ==");

I haven't fully thought out the architecture yet, but I'm certainly aiming for something along these lines.

[tlwr]

That design seems quite elegant, if datatypes are defined nicely then operations would be quite composable. How would an operation communicate the datatypes it can accept, a class variable?

e.g.

class FromBase64 extends Operation {
  INPUT_DATATYPES = ["string", "number]
}

or something more complex, e.g. support for more complicated types, although that might end up like reimplemented some XML schema...

[n1474335]

Yes, datatypes are one of the areas I haven't quite worked out yet.

We can define most of the operation config in the constructor and then generate a large config object for all operations dynamically at build time, so that the front end has all the information it requires (like operation names, descriptions and args). This dynamically built config object would replace OperationConfig.js.

class FromBase64 extends Operation {
    constructor(input, ...) {
        this.name = "From Base64";
        this.description = "Blah blah blah";
        this.categories = ["Data format"];
        this.args = ...;
        // Undecided whether args will just be the simple 'name', 'type', 'value' config used
        // currently, or a more complex but powerful object oriented model
    }
}

The theory behind this is that when contributors submit a new operation, they only have to build this one class, rather than editing about four different config files just to get the operation to appear in the UI.

Regarding datatypes... yeah. There are a number of options. We could continue having each operation support a single type which would keep this fairly simple. We could add more expressive types like File, JSON etc. which would improve the utility of certain operations such as 'Unzip' and 'Untar'.

We would then just need a good way of handling the output of the final operation in the recipe. How about there is a method in the Operation object called present() which all operations inherit. The default action would be to convert the output to a string, however certain operations could override it to add more bespoke formatting, such as the way that 'Unzip' and 'Untar' display files using HTML markup. You would then just call this method on the final operation in a recipe.

This doesn't solve the problem of translating between types when importing individual operations in a script. Any bright ideas on this subject would be welcome. The naive way of solving it would be to just expose the Dish object and the translate() method that it contains. This would prevent having to support translation within each operation class.

[mattnotmitt]

Perhaps we should make a branch with a reduced operation set to start building this? I get the feeling that it's going to be a fair bit of work to get this on track.

[n1474335]

Yes, agreed. I'll set something up once I've finished working on my current PR (not pushed yet).

It would be good to have some feedback on the new class model once the branch is set up. As you say, it's going to be a fair bit of work, so it's important we get the architecture as solid as possible from the beginning.

[n1474335]

This has been largely resolved by #284. Feel free to open a new issue if there are specifics that are outstanding.