Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make nix path-info --json return an object not array #9242

Merged
merged 4 commits into from
Nov 6, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
36 changes: 35 additions & 1 deletion doc/manual/src/release-notes/rl-next.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,39 @@

- The flake-specific flags `--recreate-lock-file` and `--update-input` have been removed from all commands operating on installables.
They are superceded by `nix flake update`.

- Commit signature verification for the [`builtins.fetchGit`](@docroot@/language/builtins.md#builtins-fetchGit) is added as the new [`verified-fetches` experimental feature](@docroot@/contributing/experimental-features.md#xp-feature-verified-fetches).

- [`nix path-info --json`](@docroot@/command-ref/new-cli/nix3-path-info.md)
(experimental) now returns a JSON map rather than JSON list.
The `path` field of each object has instead become the key in th outer map, since it is unique.
The `valid` field also goes away because we just use null instead.

- Old way:

```json5
[
{
"path": "/nix/store/8fv91097mbh5049i9rglc73dx6kjg3qk-bash-5.2-p15",
"valid": true,
// ...
},
{
"path": "/nix/store/wffw7l0alvs3iw94cbgi1gmmbmw99sqb-home-manager-path",
"valid": false
}
]
```

- New way

```json5
{
"/nix/store/8fv91097mbh5049i9rglc73dx6kjg3qk-bash-5.2-p15": {
// ...
},
"/nix/store/wffw7l0alvs3iw94cbgi1gmmbmw99sqb-home-manager-path": null,
}
```

This makes it match `nix derivation show`, which also maps store paths to information.
64 changes: 64 additions & 0 deletions src/libstore/nar-info.cc
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,15 @@

namespace nix {

GENERATE_CMP_EXT(
,
NarInfo,
me->url,
me->compression,
me->fileHash,
me->fileSize,
static_cast<const ValidPathInfo &>(*me));

NarInfo::NarInfo(const Store & store, const std::string & s, const std::string & whence)
: ValidPathInfo(StorePath(StorePath::dummy), Hash(Hash::dummy)) // FIXME: hack
{
Expand Down Expand Up @@ -125,4 +134,59 @@ std::string NarInfo::to_string(const Store & store) const
return res;
}

nlohmann::json NarInfo::toJSON(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this method used anywhere?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah right, I was going to write a round-trip unit test but didn't yet :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added unit tests!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's all nice to have it tested, but it's still dead code, isn't it? 馃檭

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but I think it is still worth it. This is the one way to test that the JSON format covers all the fields --- and it's how I discovered the compression field was missing.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I have a few ways in mind I think we might eventually use this, but I would have that as part of the motivation. Future usability is indeed not a good enough reason to add dead code.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we were to make a strict rule out of this, we'd have to move it into the test directory, but that just makes it harder to eventually use it. I think writing these is a good practice.

const Store & store,
bool includeImpureInfo,
HashFormat hashFormat) const
{
using nlohmann::json;

auto jsonObject = ValidPathInfo::toJSON(store, includeImpureInfo, hashFormat);

if (includeImpureInfo) {
if (!url.empty())
jsonObject["url"] = url;
if (!compression.empty())
jsonObject["compression"] = compression;
if (fileHash)
jsonObject["downloadHash"] = fileHash->to_string(hashFormat, true);
if (fileSize)
jsonObject["downloadSize"] = fileSize;
}

return jsonObject;
}

NarInfo NarInfo::fromJSON(
const Store & store,
const StorePath & path,
const nlohmann::json & json)
{
using nlohmann::detail::value_t;

NarInfo res {
ValidPathInfo {
path,
UnkeyedValidPathInfo::fromJSON(store, json),
}
};

if (json.contains("url"))
res.url = ensureType(valueAt(json, "url"), value_t::string);

if (json.contains("compression"))
res.compression = ensureType(valueAt(json, "compression"), value_t::string);

if (json.contains("downloadHash"))
res.fileHash = Hash::parseAny(
static_cast<const std::string &>(
ensureType(valueAt(json, "downloadHash"), value_t::string)),
std::nullopt);

if (json.contains("downloadSize"))
res.fileSize = ensureType(valueAt(json, "downloadSize"), value_t::number_integer);

return res;
}

}
11 changes: 11 additions & 0 deletions src/libstore/nar-info.hh
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,18 @@ struct NarInfo : ValidPathInfo
NarInfo(const ValidPathInfo & info) : ValidPathInfo(info) { }
NarInfo(const Store & store, const std::string & s, const std::string & whence);

DECLARE_CMP(NarInfo);

std::string to_string(const Store & store) const;

nlohmann::json toJSON(
const Store & store,
bool includeImpureInfo,
HashFormat hashFormat) const override;
static NarInfo fromJSON(
const Store & store,
const StorePath & path,
const nlohmann::json & json);
};

}
39 changes: 37 additions & 2 deletions src/libstore/parsed-derivations.cc
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,41 @@ bool ParsedDerivation::useUidRange() const

static std::regex shVarName("[A-Za-z_][A-Za-z0-9_]*");

/**
* Write a JSON representation of store object metadata, such as the
* hash and the references.
*/
static nlohmann::json pathInfoToJSON(
Store & store,
const StorePathSet & storePaths)
{
nlohmann::json::array_t jsonList = nlohmann::json::array();

for (auto & storePath : storePaths) {
auto info = store.queryPathInfo(storePath);

auto & jsonPath = jsonList.emplace_back(
info->toJSON(store, false, HashFormat::Base32));

// Add the path to the object whose metadata we are including.
jsonPath["path"] = store.printStorePath(storePath);

jsonPath["valid"] = true;

jsonPath["closureSize"] = ({
uint64_t totalNarSize = 0;
StorePathSet closure;
store.computeFSClosure(info->path, closure, false, false);
for (auto & p : closure) {
auto info = store.queryPathInfo(p);
totalNarSize += info->narSize;
}
totalNarSize;
});
}
return jsonList;
}

std::optional<nlohmann::json> ParsedDerivation::prepareStructuredAttrs(Store & store, const StorePathSet & inputPaths)
{
auto structuredAttrs = getStructuredAttrs();
Expand All @@ -152,8 +187,8 @@ std::optional<nlohmann::json> ParsedDerivation::prepareStructuredAttrs(Store & s
StorePathSet storePaths;
for (auto & p : *i)
storePaths.insert(store.parseStorePath(p.get<std::string>()));
json[i.key()] = store.pathInfoToJSON(
store.exportReferences(storePaths, inputPaths), false, true);
json[i.key()] = pathInfoToJSON(store,
store.exportReferences(storePaths, inputPaths));
}
}

Expand Down
93 changes: 93 additions & 0 deletions src/libstore/path-info.cc
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
#include <nlohmann/json.hpp>

#include "path-info.hh"
#include "store-api.hh"
#include "json-utils.hh"

namespace nix {

Expand Down Expand Up @@ -144,4 +147,94 @@ ValidPathInfo::ValidPathInfo(
}, std::move(ca).raw);
}


nlohmann::json UnkeyedValidPathInfo::toJSON(
const Store & store,
bool includeImpureInfo,
HashFormat hashFormat) const
{
using nlohmann::json;

auto jsonObject = json::object();

jsonObject["narHash"] = narHash.to_string(hashFormat, true);
jsonObject["narSize"] = narSize;

{
auto& jsonRefs = (jsonObject["references"] = json::array());
for (auto & ref : references)
jsonRefs.emplace_back(store.printStorePath(ref));
}

if (ca)
jsonObject["ca"] = renderContentAddress(ca);

if (includeImpureInfo) {
if (deriver)
jsonObject["deriver"] = store.printStorePath(*deriver);

if (registrationTime)
jsonObject["registrationTime"] = registrationTime;

if (ultimate)
jsonObject["ultimate"] = ultimate;

if (!sigs.empty()) {
for (auto & sig : sigs)
jsonObject["signatures"].push_back(sig);
}
}

return jsonObject;
}

UnkeyedValidPathInfo UnkeyedValidPathInfo::fromJSON(
const Store & store,
const nlohmann::json & json)
{
using nlohmann::detail::value_t;

UnkeyedValidPathInfo res {
Hash(Hash::dummy),
};

ensureType(json, value_t::object);
res.narHash = Hash::parseAny(
static_cast<const std::string &>(
ensureType(valueAt(json, "narHash"), value_t::string)),
std::nullopt);
res.narSize = ensureType(valueAt(json, "narSize"), value_t::number_integer);

try {
auto & references = ensureType(valueAt(json, "references"), value_t::array);
for (auto & input : references)
res.references.insert(store.parseStorePath(static_cast<const std::string &>
(input)));
} catch (Error & e) {
e.addTrace({}, "while reading key 'references'");
throw;
}

if (json.contains("ca"))
res.ca = ContentAddress::parse(
static_cast<const std::string &>(
ensureType(valueAt(json, "ca"), value_t::string)));

if (json.contains("deriver"))
res.deriver = store.parseStorePath(
static_cast<const std::string &>(
ensureType(valueAt(json, "deriver"), value_t::string)));

if (json.contains("registrationTime"))
res.registrationTime = ensureType(valueAt(json, "registrationTime"), value_t::number_integer);

if (json.contains("ultimate"))
res.ultimate = ensureType(valueAt(json, "ultimate"), value_t::boolean);

if (json.contains("signatures"))
res.sigs = valueAt(json, "signatures");

return res;
}

}
12 changes: 12 additions & 0 deletions src/libstore/path-info.hh
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,18 @@ struct UnkeyedValidPathInfo
DECLARE_CMP(UnkeyedValidPathInfo);

virtual ~UnkeyedValidPathInfo() { }

/**
* @param includeImpureInfo If true, variable elements such as the
* registration time are included.
*/
virtual nlohmann::json toJSON(
const Store & store,
bool includeImpureInfo,
HashFormat hashFormat) const;
static UnkeyedValidPathInfo fromJSON(
const Store & store,
const nlohmann::json & json);
};

struct ValidPathInfo : UnkeyedValidPathInfo {
Expand Down