Speed up multidim-interop test runner

[MarcoPolo]

In CI (using GitHub runners), each test (starting docker compose until it exits) takes about 2s. It’s unclear why this takes so long. My hunch is that it’s not the actual test that takes a while (the handshake and ping time is around 50-300ms), but the cost to setup the cgroups, network namespaces, and other docker specific things. Running the simplest docker image of hello-world takes around 400ms on my machine (time docker run --rm hello-world). There must be something here we can optimize.

Increasing the number of parallel compose stacks does not help (It does however help a bit on bigger machines).

[MarcoPolo]

Marking this as "good first issue" since it's pretty isolated.

[thomaseizinger]

Increasing the number of parallel compose stacks does not help (It does however help a bit on bigger machines).

You mean, increasing the number of WORKER_TASKS?

[MarcoPolo]

Increasing the number of parallel compose stacks does not help (It does however help a bit on bigger machines).

You mean, increasing the number of WORKER_TASKS?

yep

[thomaseizinger]

I just came across https://depot.dev/ which might be worth looking at for optimising the initial build of containers.

[MarcoPolo]

One idea would be to manually unpack the images and manually chroot into them with user namespaces. This should significantly speed things up for our use case.

A quick comparison:

Stock docker:

❯ time docker run  hello-world
// ... output ommited                                                                                                                                                                                                    
docker run hello-world  0.01s user 0.01s system 3% cpu 0.832 total

800ms!

Manually:

❯ time unshare --user --map-root-user chroot ./unpacked /hello                                                                                                                                                                                                
// ... output ommited
unshare --user --map-root-user chroot ./all /hello  0.00s user 0.00s system 58% cpu 0.007 total

7ms. 100x faster!

Of course this isn't everything Docker does, but it might be enough for our use case.

[thomaseizinger]

7ms. 100x faster!

Of course this isn't everything Docker does, but it might be enough for our use case.

Wow, that is insane! For the ping example, all we need is to run binaries right? How long does it take to unpack the images though? I wouldn't be surprised if tar is consuming a fair amount of those 800ms.

[MarcoPolo]

I wouldn't be surprised if tar is consuming a fair amount of those 800ms.

Yeah, although even after creating the container (which should cause the untarring) this still takes a bit more time.

---

I think before going to full manual route, we should explore managing the containers manually with docker directly instead of with compose. This would let us re-use the containers and only pay for container creation once for each implementation. This would greatly speed up tests that involve js-libp2p since those are pretty big containers.