Simon Willison evaluation

[progrium]

SimonW prompted an eval of webframe and got this report:

https://github.com/simonw/research/tree/main/webframe-evaluation#readme

It focuses on using it for scraping and sandboxing. Scraping is not a use case I considered, but seems like a pretty good one to support. It identifies what needs to be changed to support this. Pretty straightforward.

With sandboxing, I did intend it to be used as a sort of sandbox. I keep claiming that (in general) a browser tab or webview has been the most reliable, widely deployed sandbox we already have. This report shows what's needed for it to be a "real" sandbox, though as it points out, you can add better isolation around it. I'm hesitant to go "all the way" because the more complete a sandbox solution is, the more complex it becomes. There are easy wins, but in general, I think if you want to have "true" isolation it will probably involve multiple layers anyway. Nothing is totally secure but the more layers you have the harder it is to break.

I haven't decided what yet, and would love to hear your thoughts, but I think adding some of the easy wins focusing on the webview barrier that maintain the overall simplicity, and documenting solutions for the rest is where I'm thinking of going.

[ajbouh]

A great sandbox story seems very valuable, both to protect the machine from what webframe might do, and to allow the invoker to decide what kinds of things the loaded content might be able to do (use javascript, connect to other hosts, launch workers, etc).

I also think that a macos sandbox-exec profile and linux landlock policy are both pretty cheap and probably worth the effort.

[progrium]

Seatbelt and Landlock are exactly the extra level I was hoping to leave out. Instead focusing on hardening the webview boundary. Each platform is different and Linux is a bit more complex. Windows too. But I'm open to it. Want to hear more feedback.

[ajbouh]

Yeah, it would be good if we could count on the browser to do it for us instead.

[progrium]

Another thing I noticed is that memory usage is actually not that different than if we continued to use Chrome. The original version used the rod package, which I was introduced to by Simon's rodney CLI tool. The rod package is a high level alternative to existing CDP packages that can programmatically control Chrome. One surprising feature is it will automatically download Chromium if you do not have an existing Chrome to use (though my initial version didnt afford you to use an existing Chrome, it would always download Chromium on first use).

I have mixed feelings about this. It's nice to have a consistent platform vs native webviews, but I was always attracted to the operational simplicity of most likely not having to install a large dependency like Chrome, which sometimes introduces its own pre-reqs. This is where I also start having concerns about the rod package. We used only a handful of functions from its top level API. Yet it includes several sister dependencies, and these are decent sized packages. Plus it downloads some random version of Chrome?? There's just too much supply chain attack surface area.

THAT SAID, and this will get harder adding more options, I keep wondering if webframe should ship an alternative build that does use Chrome. Maybe not through rod, maybe some stripped down version of it. Curious to hear thoughts on this as well.

EDIT: Another reason for this is that Chromium will already apply many of the platform specific isolation technologies, so is possibly a cheap path to get that extra isolation.

[progrium]

Also towards the end, it wraps it in a script for better sandboxing. Exactly! Your sandboxing requirements will vary. For some it could be satisfied running it in Docker, such as the cgroup limits it mentions.

Long-lived worker is how this concept started back in 2015 or something. A Docker container with headless Chrome inside, with web requests routed into tabs. I still like it, but its another sort of "your requirements will vary" that makes me want to leave it out. You can easily make your own with the library or as a wrapper.

The isolated script world is a good point. That would be nice, but might be tricky to do without adding another pageload to boot time (since iframe is first thing that comes to mind).

[progrium]

Already pulled in most of the patch (not bubblewrap) and made issues for a lot of the rest. The eval did not discover Linux and Windows were being throttled, but @OEvgeny did. We got Linux fixed (afaik), next, Windows: #2