Memory Use Issues for Large Store Paths

[blitz]

A user reports large store paths to cause the client to run out of memory: https://discourse.nixos.org/t/celler-an-attic-fork/77265/19

[greg-hellings]

@blitz I can't pretend to know everything about what is going on here, as I don't know this code very much, but this is the patch I maintain to allow Attic Client to handle large files. I'm not ashamed to admit that some LLM help went into creating this and finding the memory bottleneck. Apparently the file was being entirely read into memory previously before sending, rather than the file being streamed to the server. The files I was working with included 2 or 3 which exceeded 100GB in size, while my laptop has only 64GB of RAM. So this was causing the client to crash.

This change seems to have worked properly to allow me to upload files. The speed issues I mentioned in the thread are still present and pervasive. I can't seem to exceed about 10MB/s transfers into and out of the cache. For most small artifacts this hardly is a problem as latency is the limiting factor there. But when retrieving those 100GB+ files, or even a larger normal package like qemu, firefox, etc then the speed issues become readily apparent. I don't have, and haven't pursued, a triaging of that as it is not preventing me from using Attic. It just makes doing so a bit tedious.

[blitz]

I see the issue. There is the unbounded channel. That really helped. Thanks!

[blitz]

Hey @greg-hellings! The offending code doesn't exist anymore in celler. Memory use for pushing large paths is constant for me (~65MB), which seems reasonable.