Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upstream binary cache #16

Closed
domenkozar opened this issue May 30, 2018 · 16 comments
Closed

Upstream binary cache #16

domenkozar opened this issue May 30, 2018 · 16 comments
Assignees
Milestone

Comments

@domenkozar
Copy link
Member

@domenkozar domenkozar commented May 30, 2018

There is a common use case why a binary cache would declare other caches as upstream: It expects binaries to come from another binary cache: it is true for all cachix binary caches to depend on official cache.nixos.org

Possible solutions:

a) "Proxy": if there is no such narinfo in current cache, query upstream caches and redirect. Possible issue here is how Nix handles signatures to match the binary cache declared, but I think it could work.

b) "Resign": currently not possible as signing key is not available to cachix and I'd prefer not to support such option.

Brought up by @zimbatm in #15 (comment)

@Zimmi48
Copy link

@Zimmi48 Zimmi48 commented Jun 23, 2018

Indeed, I'd be glad if I could avoid re-pushing to my personal cachix everything that is already in https://cache.nixos.org.

@dtzWill
Copy link

@dtzWill dtzWill commented Jun 23, 2018

For cachix devs/provider a similar issue/idea would be to dedup across cachix's in a manner similar to what nix-store --optimize does. I suppose there's also the question of whether this would be undesirable, same for when using an upstream, since the redundancy might in some cases be desired/intentional.

@asymmetric
Copy link
Contributor

@asymmetric asymmetric commented Jun 23, 2018

I suppose there's also the question of whether this would be undesirable, same for when using an upstream, since the redundancy might in some cases be desired/intentional.

This could be a local setting.

@chris-martin
Copy link

@chris-martin chris-martin commented Aug 4, 2018

I think I'd be utilizing cachix a lot more if we had this feature. I'd like to build NixOS on my desktop, push to cachix, then build on my laptop, so that the laptop doesn't have to rebuild any of my non-standard packages. But in practice right now this doesn't actually save me any time, because it takes me quite a while (I don't know, maybe an hour?) to push my NixOS build to cachix -- which is silly because most of what I'm pushing is already in the upstream cache.

@domenkozar
Copy link
Member Author

@domenkozar domenkozar commented Aug 5, 2018

I wouldn't recommend pushing laptop paths that include also passwords and such to a public cache. Although it's obscured by someone needing to know the hash, if you post it somewhere by accident, they could get your /etc/passwd for example.

About the upstream binary cache: I think this would be set as a property to a binary cache where by default it would be cache.nixos.org, but one could remove it or specify additional ones.

@chris-martin
Copy link

@chris-martin chris-martin commented Aug 5, 2018

I've set my user passwords with passwd and don't have any secrets in the NixOS configuration. So that shouldn't be a concern, right?

@domenkozar
Copy link
Member Author

@domenkozar domenkozar commented Aug 5, 2018

Some Nix modules store secrets in Nix store, it depends on per use case. It's generally bad practice, but you do find it in the wild.

@domenkozar domenkozar added this to the 0.2 milestone Sep 26, 2018
@domenkozar domenkozar removed this from the 0.2 milestone Jan 18, 2019
@domenkozar domenkozar added this to the 0.3 milestone Jan 18, 2019
@risicle
Copy link

@risicle risicle commented Apr 22, 2019

Something as simple as a cachix push mycache --exclude-present-in https://cache.nixos.org option would do the job for me. This seriously restricts the usefulness of cachix for me and I'm sure fills your server with duplicates.

@domenkozar
Copy link
Member Author

@domenkozar domenkozar commented Mar 2, 2020

Going to implement this in a few weeks.

@domenkozar domenkozar self-assigned this Mar 2, 2020
@domenkozar
Copy link
Member Author

@domenkozar domenkozar commented Jun 8, 2020

Sorry, the priority shifted a bit. This is now my 6th item in the backlog, so 2-3 weeks.

@lf-
Copy link

@lf- lf- commented Jul 10, 2020

Hi @domenkozar what's the status on this feature right now? Is there a viable workaround? I have a /pile/ of build time dependencies that I really would like to avoid forcing anyone else to build as it takes over an hour to build all of them. This is also going to become a CI problem to compile so much Haskell that's not mine ;-)

@domenkozar
Copy link
Member Author

@domenkozar domenkozar commented Jul 10, 2020

I'm currently working on https://docs.cachix.org/ to cover different CI integrations and general information people are missing right now, and after that it's this issue.

There is no workaround really at the moment. The overhead of pushing to Cachix shouldn't be that much, are you recompiling dependencies often? Once they are pushed once, they should just stay there.

@domenkozar
Copy link
Member Author

@domenkozar domenkozar commented Jul 21, 2020

Starting the work on this today.

@domenkozar
Copy link
Member Author

@domenkozar domenkozar commented Jul 28, 2020

@domenkozar domenkozar closed this Jul 28, 2020
@adrian-gierakowski
Copy link

@adrian-gierakowski adrian-gierakowski commented Jul 28, 2020

@domenkozar
Copy link
Member Author

@domenkozar domenkozar commented Jul 28, 2020

No. Implementing it on server side has quite a few advantages:

  • global cache of known entries
  • enforced for the whole cache
  • quite likely faster as it holds open connection all the time

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants