Rewrite fetchgitLocal

[Ericson2314]

For practical purposes, here are the changes in behavior:

• When fetching from a subdirectory of a repo, do not rebuild because of
  changes elsewhere in the repo
• Fetch (not-ignored) untracked files too

It does this by letting git hash and export the directory in question, which I believes makes for a cleaner implementation than the ad-hoc copying and hashing that was there before.

We have multiple library in our repo (see https://github.com/unisonweb/platform/blob/master/env.nix) and it was very annoying to needless rebuild everything if anything changes.

@pchiusano

[gridaphobe]

This git add will be reversed by the mv command on line 24, correct?

[Ericson2314]

Exactly.

[puffnfresh]

I like it 👍

[Ericson2314]

Thanks!

[gridaphobe]

👍 thanks!

[edolstra]

Eh. I'm not in favor of using builtins.currentTime in Nixpkgs. It's an undocumented misfeature that shouldn't be relied on.

[gridaphobe]

@edolstra is there a better alternative?

This use of builtins.currentTime is essentially a hack to mark the computed hash as impure, so nix will recompute it every time. Without this step, nix is too clever and skips rebuilding the local package because the the source directory is the only other input and it is invariant.

Is there a way to mark expressions as impure? That's what we really want to do for the hash computation.

[Ericson2314]

@edolstra First of all, the old version used the same basic trick, I did not introduce it.

The idea, (before and after my changes) is the first derivation is impure, and generates a "token" used syntactically (via a dynamic import, readFile, etc). The second derivation, taking that "token" as a syntactic input (i.e. we have a dynamic import or readFile, both of which are monadic bind-like constructs) is thus pure---ideally the token would be a fixed-output hash to "proove" the purity.

I'd say the root problem is our treatment of local paths. It's impossible to write a derivation that relies on paths outside of the nix-store and isn't a fixed out, and expect it to be deterministic. Our current trick of copying local paths to the store amounts to the same thing as this fetch git local stuff: an initial impure step carefully constructed so as not to invalidate caching downstream.

Perhaps we ought to generalize this with impure derivations: they get built every time, but are hashed according to their contents not their inputs so as not to affect downstream. Instead of always copying local paths to the store, hopefully sandboxed builds can be the default so such paths just aren't accessible at all.

Note the once we have an intentional store, the content-hashing part of such impure derivations would simply fall out of the normal rules. Also, everything here could in fact be done in one big impure derivation, and downstream would still be cached just as well.

[Ericson2314]

@edolstra Also, this could be rewritten with a src = ./. to copy the entire git repo, and then extract the correct directory, which I suppose is the conventional way of doing such things today. But that would be much inferior. First of all, since the git repo contains all the history, it can be be indefinitely bigger than the working copy. Second of all, the hash then tracks a bunch of irrelevant other state and thus one gets many more spurious rebuilds.

[gridaphobe]

@Ericson2314 that's exactly why I wrote fetchgitLocal in the first place. Using src = ./. causes needless recompilation due to build artifacts, and can also be extremely slow since it has to hash all of the build artifacts.

[Ericson2314]

Sure! Just want to make why things are the way they are for @edolstra or anyone else that is interested.