Uploading to cachix is slow, it seems to reupload stuff even when not needed

[arximboldi]

Hi!

I use cachix with a paid plan and Gibhub Actions and I love it. Particularly on the infra CI, managed via Colmena, it is important.

However, I've noticed that even in no-op builds where it downloads everything from cache (the build itself takes about 10mins, because it needs to download a fair amount of stuff), it seems to still try to reupload 7 GB of data, which takes much longer, around 40min.

Maybe something like #86 would already make things better. But maybe there is something wrong that is making the upload particularly slow, or making cachix not realize that it doesn't really need to reupload anything.

Some suggestions about what is going on? Is this maybe a bug?

[arximboldi]

For some reason, it is always these 3 derivations that get pushed... I have looked into them, and those are some large derivations containing some data-files that are downloaded with fetchzip. Our project contains other similar derivations that are correctly cached, downloaded from cachix in subsequent builds, and not re-pushed every single time. Maybe it is a temporary glitch and this will fix itself after a few runs.

Interestingly, these derivations could maybe be directly ignored. It seems however, looking into issues around exclusion rules in the cachix repo, that this is not an easy thing to do...?

[domenkozar]

I'd suggest making -source derivations smaller in the first place. Everytime you evaluate your project, it will copy that many gigabytes into your nix store, duplicating all data.

[arximboldi]

Not sure if that is possible. However, it seems the issue has gotten better over time. Sorry for the noise.