Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/libstore-c/nix_api_store.cc
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C API could be extended when the need arises. Now seems premature, so this impl-only change looks perfect to me.

Original file line number Diff line number Diff line change
Expand Up @@ -251,7 +251,7 @@ nix_err nix_store_copy_closure(nix_c_context * context, Store * srcStore, Store
if (context)
context->last_err_code = NIX_OK;
try {
nix::RealisedPath::Set paths;
nix::StorePathSet paths;
paths.insert(path->path);
nix::copyClosure(*srcStore->ptr, *dstStore->ptr, paths);
}
Expand Down
3 changes: 2 additions & 1 deletion src/libstore/include/nix/store/legacy-ssh-store.hh
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,8 @@ public:
StorePathSet & out,
bool flipDirection = false,
bool includeOutputs = false,
bool includeDerivers = false) override;
bool includeDerivers = false,
std::function<bool(const StorePath & path)> pathCallback = nullptr) override;

StorePathSet queryValidPaths(const StorePathSet & paths, SubstituteFlag maybeSubstitute = NoSubstitute) override;

Expand Down
8 changes: 7 additions & 1 deletion src/libstore/include/nix/store/store-api.hh
Original file line number Diff line number Diff line change
Expand Up @@ -802,13 +802,19 @@ public:
* `storePath` is returned; that is, the closures under the
* `referrers` relation instead of the `references` relation is
* returned.
*
* @param onPathDiscovered A callback invoked on each discovered store
* path. Returning `false` from the callback will stop recursion on
* that path. Returning `true` will continue recursion. A `nullptr` is
* equivalent to a callback which always returns `true`.
*/
virtual void computeFSClosure(
const StorePathSet & paths,
StorePathSet & out,
bool flipDirection = false,
bool includeOutputs = false,
bool includeDerivers = false);
bool includeDerivers = false,
std::function<bool(const StorePath & path)> onPathDiscovered = nullptr);

void computeFSClosure(
const StorePath & path,
Expand Down
11 changes: 8 additions & 3 deletions src/libstore/legacy-ssh-store.cc
Original file line number Diff line number Diff line change
Expand Up @@ -252,10 +252,15 @@ void LegacySSHStore::buildPaths(
}

void LegacySSHStore::computeFSClosure(
const StorePathSet & paths, StorePathSet & out, bool flipDirection, bool includeOutputs, bool includeDerivers)
const StorePathSet & paths,
StorePathSet & out,
bool flipDirection,
bool includeOutputs,
bool includeDerivers,
std::function<bool(const StorePath & path)> onPathDiscovered)
{
if (flipDirection || includeDerivers) {
Store::computeFSClosure(paths, out, flipDirection, includeOutputs, includeDerivers);
if (flipDirection || includeDerivers || onPathDiscovered) {
Store::computeFSClosure(paths, out, flipDirection, includeOutputs, includeDerivers, onPathDiscovered);
return;
}

Expand Down
9 changes: 8 additions & 1 deletion src/libstore/misc.cc
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ void Store::computeFSClosure(
StorePathSet & paths_,
bool flipDirection,
bool includeOutputs,
bool includeDerivers)
bool includeDerivers,
std::function<bool(const StorePath & path)> onPathDiscovered)
{
std::function<std::set<StorePath>(const StorePath & path, std::future<ref<const ValidPathInfo>> &)> queryDeps;
if (flipDirection)
Expand Down Expand Up @@ -65,6 +66,12 @@ void Store::computeFSClosure(
startPaths,
paths_,
[&](const StorePath & path, std::function<void(std::promise<std::set<StorePath>> &)> processEdges) {
if (onPathDiscovered && !onPathDiscovered(path)) {
std::promise<std::set<StorePath>> promise;
promise.set_value({});
processEdges(promise);
return;
}
std::promise<std::set<StorePath>> promise;
std::function<void(std::future<ref<const ValidPathInfo>>)> getDependencies =
[&](std::future<ref<const ValidPathInfo>> fut) {
Expand Down
13 changes: 10 additions & 3 deletions src/libstore/store-api.cc
Original file line number Diff line number Diff line change
Expand Up @@ -1072,9 +1072,16 @@ void copyClosure(
if (&srcStore == &dstStore)
return;

StorePathSet closure;
srcStore.computeFSClosure(storePaths, closure);
copyPaths(srcStore, dstStore, closure, repair, checkSigs, substitute);
StorePathSet pathsToCopy;

auto onPathDiscovered = [&](const StorePath & path) -> bool {
// Only recurse if the path does not already exist in `dstStore`
return repair || !dstStore.isValidPath(path);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this line is what @Ericson2314 had concerns about. If the destination store is something like ssh:// or ssh-ng:// this can have a large latency. copyPaths does a queryValidPaths that does the queries in parallel as a hack around this issue. Ideally computeFSClosure was itself async and the callback could be a coroutine, then computeFSClosure could continue processing other paths while it's awaiting the path info. This reminds me that we badly need a coroutine-based async framework for store operations, but that's a much bigger long-term goal.

};
srcStore.computeFSClosure(storePaths, pathsToCopy, false, false, false, onPathDiscovered);
pathsToCopy.insert(storePaths.begin(), storePaths.end());

copyPaths(srcStore, dstStore, pathsToCopy, repair, checkSigs, substitute);
}

std::optional<ValidPathInfo>
Expand Down
Loading