Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Git hashing as a new file ingestion method --- contains #3754 #3635

Closed
wants to merge 231 commits into from

Conversation

Ericson2314
Copy link
Member

@Ericson2314 Ericson2314 commented May 28, 2020

This is an umbrella PR for all the work we are doing towards git object support in Nix, as part of https://discourse.nixos.org/t/obsidian-systems-is-excited-to-bring-ipfs-support-to-nix . We will probably break up into smaller PRs as it nears completion. More details forthcoming.


CC @nlewo

depends on #3754

@grahamc
Copy link
Member

grahamc commented May 28, 2020

Interesting! I wonder if we could teach IPFS about NARs, in the same way that IPFS knows about Git?

@Ericson2314
Copy link
Member Author

Ericson2314 commented May 28, 2020

@grahamc We could, no doubt about it. But there are reasons to want to try git objects even if it is not the only way to talk to IPFS:

We'll write an RFC that lays out a fully-fleshed motivation. This PR is supposed to be the sneak preview / if anyone has any specific implementation concerns we ought to know about sooner rather than later.

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/obsidian-systems-is-excited-to-bring-ipfs-support-to-nix/7375/17

@zimbatm
Copy link
Member

zimbatm commented May 30, 2020

@Ericson2314 are there limitations around large files similar to Git? Is it possible to cut them up in smaller pieces in a casync fashion?

@Ericson2314
Copy link
Member Author

Ericson2314 commented May 30, 2020

@zimbatm git works with arbitrary sized files, but poorly. I went with it because it is popular, especially among the data we have as input (git repos, of course)---it's important for network effects that the addresses themselves, not merely the data, be well known.

I think that's a fine choice for now, and in a world where these p2p technologies are much more widely used we'll have ample opportunity to change it. #3640 for example doesn't say we should put git and nar on equal footing, but anything that might go in the ca field and nar. Git itself will need to migrate off sha-1, and while there is a plan already (https://github.com/git/git/blob/master/Documentation/technical/hash-function-transition.txt), perhaps there is still time to raise large file / chunking issues.

Ericson2314 and others added 13 commits June 1, 2020 16:03
Hopefully we can instead pass arguments like Hydra does, soon.
This matches what we want for blobs.

Trees are still in progress - we need a way to symlink to other
objects, using that to determine ca.
This is really bad and dangerous! But Git migration to sha256 is still
a ways away:

https://lwn.net/Articles/811068/

So we need to allow it for the time being.
We need access to other things in the store. This is kind of dangerous
though if things are added in the wrong order�.
This is needed to create files based on git permissions
This updates the remote protocol to try to handle sha1 hashes.
@github-actions github-actions bot added new-cli Relating to the "nix" command with-tests Issues related to testing. PRs with tests have some priority store Issues and pull requests concerning the Nix store fetching Networking with the outside (non-Nix) world, input locking labels Sep 4, 2023
@Ericson2314
Copy link
Member Author

Closing in favor of #8918. What basically happened is that think the other order, Git hashing first, then content-addresed store lookup, makes more sense. So if that gets merged this, this will become the same as #3754

@Ericson2314 Ericson2314 closed this Sep 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fetching Networking with the outside (non-Nix) world, input locking new-cli Relating to the "nix" command store Issues and pull requests concerning the Nix store with-tests Issues related to testing. PRs with tests have some priority
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants