Permalink
Browse files

Add fetchgit builtin

The function builtins.fetchgit fetches Git repositories at evaluation
time, similar to builtins.fetchTarball. (Perhaps the name should be
changed, being confusing with respect to Nixpkgs's fetchgit function,
with works at build time.)

Example:

  (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello

or

  (import (builtins.fetchgit {
    url = git://github.com/NixOS/nixpkgs-channels;
    rev = "nixos-16.03";
  }) {}).hello

Note that the result does not contain a .git directory.
  • Loading branch information...
edolstra committed Apr 29, 2016
1 parent 8325822 commit 38539b943a060d9cdfc24d6e5d997c0885b8aa2f
Showing with 78 additions and 1 deletion.
  1. +77 −0 src/libexpr/primops/fetchgit.cc
  2. +1 −1 src/libstore/download.cc
@@ -0,0 +1,77 @@
#include "primops.hh"
#include "eval-inline.hh"
#include "download.hh"
#include "store-api.hh"
namespace nix {
static void prim_fetchgit(EvalState & state, const Pos & pos, Value * * args, Value & v)
{
// FIXME: cut&paste from fetch().
if (state.restricted) throw Error("‘fetchgit’ is not allowed in restricted mode");
std::string url;
std::string rev = "master";
state.forceValue(*args[0]);
if (args[0]->type == tAttrs) {
state.forceAttrs(*args[0], pos);
for (auto & attr : *args[0]->attrs) {
string name(attr.name);
if (name == "url")
url = state.forceStringNoCtx(*attr.value, *attr.pos);
else if (name == "rev")
rev = state.forceStringNoCtx(*attr.value, *attr.pos);
else
throw EvalError(format("unsupported argument ‘%1%’ to ‘fetchgit’, at %3%") % attr.name % attr.pos);
}
if (url.empty())
throw EvalError(format("‘url’ argument required, at %1%") % pos);
} else
url = state.forceStringNoCtx(*args[0], pos);
if (!isUri(url))
throw EvalError(format("‘%s’ is not a valid URI, at %s") % url % pos);
Path cacheDir = getCacheDir() + "/nix/git";
if (!pathExists(cacheDir)) {
createDirs(cacheDir);
runProgram("git", true, { "init", "--bare", cacheDir });
}
Activity act(*logger, lvlInfo, format("fetching Git repository ‘%s’") % url);
std::string localRef = "pid-" + std::to_string(getpid());
Path localRefFile = cacheDir + "/refs/heads/" + localRef;
runProgram("git", true, { "-C", cacheDir, "fetch", url, rev + ":" + localRef });
std::string commitHash = chomp(readFile(localRefFile));
unlink(localRefFile.c_str());
debug(format("got revision ‘%s’") % commitHash);
// FIXME: should pipe this, or find some better way to extract a
// revision.
auto tar = runProgram("git", true, { "-C", cacheDir, "archive", commitHash });
Path tmpDir = createTempDir();
AutoDelete delTmpDir(tmpDir, true);
runProgram("tar", true, { "x", "-C", tmpDir }, tar);
Path storePath = state.store->addToStore("git-export", tmpDir);
mkString(v, storePath, PathSet({storePath}));
}
static RegisterPrimOp r("__fetchgit", 1, prim_fetchgit);
}
@@ -313,7 +313,7 @@ bool isUri(const string & s)
size_t pos = s.find("://");
if (pos == string::npos) return false;
string scheme(s, 0, pos);
return scheme == "http" || scheme == "https" || scheme == "file" || scheme == "channel";
return scheme == "http" || scheme == "https" || scheme == "file" || scheme == "channel" || scheme == "git";
}

5 comments on commit 38539b9

@domenkozar

This comment has been minimized.

Member

domenkozar replied Apr 30, 2016

We really need to be consistent here and pick a name that:

  • hints the fetching is done at eval time
  • matches fetchTarball alternative that just uses a different format
@shlevy

This comment has been minimized.

Member

shlevy replied May 2, 2016

nix-exec has a fetchgit builtin we may want to pull from here https://github.com/shlevy/nix-exec/blob/master/src/fetchgit.cc https://github.com/shlevy/nix-exec/blob/master/scripts/fetchgit.sh.in

FWIW I'm a bit worried about using one local cache repo for everything. It doesn't seem to gain much (seems unlikely that a lot of big files will be shared between repos) and just costs us a higher chance of collision. nix-exec has a separate local repo for each repo.

@lheckemann

This comment has been minimized.

Member

lheckemann replied Mar 13, 2018

@shlevy I think significant savings can be made when sharing e.g. nixpkgs and nixpkgs-channels, or different linux kernel repos, for example.

@shlevy

This comment has been minimized.

Member

shlevy replied Mar 13, 2018

@lheckemann On the other hand, we see ridiculously huge slowdowns (to the point of unusability) when using builtins.fetchGit on multiple actually-distinct repos. I'm fine with a way to give a common cache name to disparate repos (e.g. "nixpkgs-channels should live with nixpkgs"), but the current situation is effectively broken for the multi-repo usecase.

@lheckemann

This comment has been minimized.

Member

lheckemann replied Mar 13, 2018

Huh, I've never seen that. I suppose I don't use fetchGit enough!

Please sign in to comment.