Consider a WASM build

[danbri]

Noticing this is getting ported to Golang, Rust ... would it be worth integrating a WASM (web assembly) build into the process?

[AVGP]

Quick question - is the main goal to bring it to a browser or is there a specific reason it needs to be wasm?
I'm asking because, behind the scenes for the most part, I'm porting it to node.js and that should (I gotta double check that tho) also be browser-compatible when it's done.

[carlopi]

I did compile it with Cheerp, a C++ to JavaScript or JavaScript + WebAssembly compiler, for now exposing only the AllowedByRobots function:

[[cheerp::jsexport]][[cheerp::genericjs]]
bool AllowedByRobots(client::String* robots_body,
                client::TArray<client::String>* user_agents,
                client::String* url)
{
        RobotsMatcher robot;
        
        std::string body = (std::string)(*robots_body);
        std::vector<std::string> vec;
        for (int i=0; i<user_agents->get_length(); i++)
        {
                std::string S = (std::string)(*((*user_agents)[i]));
                vec.push_back(S);
        }
        std::string URL = (std::string)(*url);

        return robot.AllowedByRobots(body, &vec, URL);
}

Adding the rest of the API should be trivial (possibly with async wrappers that also take care of fetching the robot file), and the result is either a single JavaScript file (73K for the ugly version) or a combined of 46K of JavaScript + WebAssembly payload.

Setting on a proper API + some round of testing would be needed, but this should already work as expected https://gist.github.com/carlopi/d097af89855861e4cde964dbce8f4fc3 (mostly see test.js, the compiled JavaScript is not particularly informative).

Looking forward on some input on how to move forward + whether this can be of help.