Fix FS IDs growing too large

[sipsma]

We currently pass around the json serialization of LLB (wrapped w/ b64 to avoid confusion) as the FS type (or the ID of the new Filesystem type that will be replacing FS).

The problem is that this string can already grow to be quite large in simple cases (such as those involving dockerfiles). Once we have to deal with more complex cases, it may become too large and cause genuine performance problems in addition to unreadable output.

We probably need to switch to storing the full LLB server-side only and making the ID just be a hash of that. It's unfortunate because there are nice side-effects of the ID of LLB being the actual executable definition of it too, but I'm not sure of any way of retaining that property that won't eventually hit the same problem of growing too large.

[sipsma]

Thought I was going to fix this here: dagger/cloak@35b0798

But then realized there's a flaw in the change to using a hash for FSID: if a result is cached from a previous run, it can return FSIDs for steps that were skipped during the current execution, in which case we won't know the mapping between the hash and actual LLB because everything is stored only in memory. So it sometimes breaks depending on the caching of the run.

I am now wondering if this would be a good upstream change; I can imagine buildkit exposing an API that lets you retrieve cached results based on LLB digest.

EDIT: nevermind about the above idea, it wouldn't work in the case where the cache was pruned. This is actually going to be a tricky problem. We essentially want persisted state mapping hashes to LLB, but we can't really rely on buildkit's state because it is inherently just best-effort (cache could be pruned anytime).

I'd like to avoid having to maintain our own state as much as possible, but only reliable alternative I can think of at the moment is to compress the strings (i.e. with zlib), which is kind of silly and only going to delay this problem rather than fix it, so needs more thought.

[sipsma]

Fixed by dagql and subsequent ID flattening