-
-
Notifications
You must be signed in to change notification settings - Fork 121
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create translator and builder for rust/cargo #16
Comments
One package that would be worth testing is meilisearch github.com/meilisearch/meiliSearch/ A bit more context on meilisearch: I'm the maintainer of it in nixpkgs. Originally I was using crate2nix to build it, but transitioned to a more classic buildRustPackage build to simplify updates. It worked for a bit but now it's problematic again to the point I couldn't do the last update. |
Thanks for the info. |
I think we can leverage the Cargo.lock / Cargo.toml parsing from |
I have started working on the translator here: https://github.com/yusdacra/dream2nix/tree/feat/cargo-translator My main questions now are:
|
See this little chart for an overview of the architecture. There is also an example walking through the steps. Translators are expected to output the dream lock format. This format is specified by a jsonschema defined in ./src/specifications. There is also an example next to the schema. The schema is verified by the python CLI, in case the translation is triggered through the CLI . The generated translator template, you started working with, uses
Subsystem is a category so that all packages within one subsystem can be built the same way, even if the data (dream-lock.json) which defines those packages has been created by different translators. As an example, for nodejs we have different translators as there exist different lock file formats. But all data created by these translators must be buildable by the same builder.
You are a 100% right, this is a design issue which needs to be fixed. The option should be removed
Example: given this exemplary upstream lock file: {
dependencies = {
requests = {
version = "1.2.3";
url = "https://...requests.tar.gz";
hash = "deadbeef...";
};
async = {
version = "2.3.4";
url = "https://...async.tar.gz";
hash = "deadbeef2...";
};
};
} ...processed via [
{
name = "requests";
version = "1.2.3";
url = "https://...requests.tar.gz";
hash = "deadbeef...";
}
{
name = "async";
version = "2.3.4";
url = "https://...async.tar.gz";
hash = "deadbeef2...";
}
] The upstream structure was already relatively flat, so we just needed to incorporate the name into each object. Please let me know if you have more questions. |
Ok, i see. So the "getSourceType" and "sourceConstructors" need to output the source types specified in the dream.lock schema. Would it make sense to add a Also about |
It will not be required for packaging projects from github. Translating a github project will result in a list of http sources, so the
{ name = "package-name"; version = "1.2.3"; } |
I'm not sure what you meant by "packaging projects from github". I meant that regardless of whether a project is from github or crates-io, it will have dependencies from crates-io. They can be downloaded through http, but they require to be decompressed (https://github.com/nix-community/naersk/blob/master/build.nix#L433-L449). So i meant it in that sense, i suppose my question being if the builder would decompress it? Also yeah there are plenty of applications on crates.io (ripgrep, bottom, exa and many more) so i guess it would be useful. |
getting assets from npm also requires decompression. It should be fine for crates.io too. You are right about the rust applications. Having a fetcher would enable easy download of those applications. It's just like npm though, it means you have to trust the binary. |
@yusdacra I meant that only if the initial source (the main projects source) is coming from crates.io, we need a specific fetcher for that, because we need the CLI to understand references like But, as I now understand that there are applications hosted on crates.io, we need that creates.io fetcher anyways. Since b8dc44d, each fetcher in dream2nix is expected to return a decompressed source, so the decompression and the creation of the So after all it might be better to just use the
I think we are talking about package sources here, not binaries. |
Ok, that sounds good. I will write the translator so that it uses a cargo registry fetcher then. We can have a crates-io specific fetcher, but i think a general cargo registry fetcher could be better (if people want to use dream2nix against a different registry). The index format of a registry is defined here https://doc.rust-lang.org/cargo/reference/registries.html#index-format . The problem with this is we would need to clone the git repository (cargo registries are git repos) and then look for the index file to get the download API URL. This would be very inefficient. So for now it's probably best to have just a crates-io fetcher. |
Sounds good. Then let's start with the crates-io fetcher only. |
I was thinking of: {
checksum = "";
name = "";
version = "";
} for the schema format. |
Sounds good, though the Maybe a good way to start is to just copy |
I pushed some more changes to https://github.com/yusdacra/dream2nix |
Thanks I will review it tomorrow. No need to split it into several PRs. It is up to you. |
I see you made use of This way, your translator will be compatible to the current CLI. You can then test your translator by just executing for example:
This will generate the dream-lock.json for the given input. Maybe it is a bit confusing that the translator interface already allows for several files and directories being passed. Its not really used anywhere yet, but I wanted to keep the possibilities open for the future. |
Ah alright, i see. I pushed more work and tested out the translator, it seems to be passing the translation stage but dream2nix fails afterwards with:
any way I can see the JSON? Would like help on that. And also, i see that the npm fetcher doesn't do decompression, so should it be handled by builders? |
Nevermind, i fixed it. Turns out it was a trailing comma in spec. Wish this had better error reporting or maybe JSON allowed trailing commas 😅 |
It seems to generate the dream lock fine now! 🎉 And btw, is there a reason why a "main package" is needed in a dream lock? For example, a Cargo.lock can contain multiple packages that are in a workspace. So it makes more sense to choose a package to build in the builder instead of the translation process. Not sure if there is already a way to do that. |
Could you point me to a github project that uses several packages. From what I observed in nodejs so far, there usually is always a main package even if the repo contains several packages. Usually the extra packages are sub-packages of the main package.
Right now each If it is not clear which package is the main package, maybe we should allow the dream-lock to have |
Repos like https://gitlab.com/veloren/veloren have seperate binary packages in a single workspace, others like https://github.com/iced-rs/iced use seperate Cargo packages for their examples. You can't really determine a "main package" if the user doesn't pass a package name and this is a workspace where multiple (binary) packages are available. So it's probably better to just delegate that to the builder where the user can select what they want to build. I think it would be nice to allow Ideally IMO it would be a better idea to drop |
The dram2nix CLI is only used for translation, not for building as of now. And I don't see a reason why we would need it for building. As long as all packages are exposed via an attribute, the standard nix CLI should be sufficient. So I think we should just allow mainPackageName to be null or unset. |
Thats fine. What i meant was that we remove the main package from lock, the users can just use whatever package provided by Also yeah I was talking about for debugging since that generates an expression. But i suppose setting a main package doesn't really matter since you can just specify the attribute. |
OK, removing the main package sounds good. |
We could close this issue now, but I will take the opportunity to use this as a tracking issue, as I want to implement a granular builder for Rust. |
I am currently working on a feature to support multiple lock files in a single repo. This is quite common in nodejs. |
That sounds good. |
Current situation is: There is a translator for Next we want to have a granular builder, ie. a builder that will build each crate in a dependency tree as a separate derivation. For this we can use I think this issue can be closed now that we actually have translators and builders for Cargo. |
Some work on the translator has already been started but is stalled: #5
We might be able to re-use some parts of import-cargo-lock.nix from nixpkgs.
The text was updated successfully, but these errors were encountered: