Directory downloads process one-file-at-a-time

[allada]

We use bazel's directory api in for tar ball unpacking and when we use bb-remote-execution it only processes one file a time which takes ~1 hour in our case.

I built a PR to fix this, but I don't believe it's production ready:
allada@27022d6

[EdSchouten]

Yeah, right now there isn’t any parallelism in that code. Adding parallelism to it (though bounded) would speed up things significantly.

It looks like the fix you wrote for it is for a relatively old version of Buildbarn. In the latest version of the source tree it’s likely easier to implement. All of the build directory population code is now constrained to this type:

https://github.com/buildbarn/bb-remote-execution/blob/master/pkg/builder/naive_build_directory.go

Relatedly, uploads could be parallelized by extending this code:

https://github.com/buildbarn/bb-remote-execution/blob/master/pkg/blobstore/batched_store_blob_access.go

[EdSchouten]

Even though it's still the case that the native input root population is single threaded, I am going to close this issue. In my opinion nobody should spend any time trying to optimise support for native input roots. There is no need to sugar coat it.

FUSE/NFSv4 is the way to go, and the implementation we have is pretty robust at this point. We recently added support for the File System Access Cache, meaning that FUSE/NFSv4 is lazy loading and capable of prefetching files in parallel.