Add an API to generate registry index data

[carols10cents]

See RFC 2141 for more details/context.

Currently, the knowledge of what goes in the registry index in what format lives in crates.io:

• here,
• which serializes an instance of this struct,
• which comes from here,
• most of which comes from cargo anyway, around here,
• which deserializes an instance of this struct
• which has a corresponding struct in cargo
• that gets created here

In order to support minimal registries, we'd like cargo to be able to generate the JSON information that should go into a registry for the current crate.

Cargo does already understand how to read these entries! We need to teach it how to create them as well :)

So as I understand it, this would involve:

• Derive Serialize for the RegistryPackage struct
• Create a new command, feel free to suggest another name, the first thought in my head is generate-index-metadata, that should:
  • Create an instance of RegistryPackage for the current crate, reusing code that sends relevant information to crates.io when appropriate
  • Serialize that struct to stdout

[carols10cents]

In order to test out #4506, I've created a minimal registry! The part that this proposed command would create is this file's contents, given the current crate.

I don't think we envision people running this command and doing something manually with the output, though that will be possible.

The main goal is to enable any tool to be able to create the same registry entries that crates.io creates right now, without needing to run all of crates.io.

[stephanbuys]

@carols10cents from what I can tell #4506 deals with cargo support for alt-registries, not this issue (ie. generating the actual minimal registry data). I'd like to have a stab at it if this command is still up for grabs.
Does it make sense to abstract some of this work out into a 3rd party crate to be used by crates.io and other registry implementations?

[carols10cents]

@stephanbuys

@carols10cents from what I can tell #4506 deals with cargo support for alt-registries, not this issue (ie. generating the actual minimal registry data).

Yes, #4506 is only tangentially related-- someone on gitter was asking for an example of what a "minimal registry" would look like and what this command needs to generate, and #4506 had just happened to prompt me to do just that :)

Does it make sense to abstract some of this work out into a 3rd party crate to be used by crates.io and other registry implementations?

Cargo can be used as a dependency, and someone on gitter also suggested adding a public API function that returns the text this command would output, so I think that would work. Does that sound ok?

[liranringel]

I started to look into it.
RegistryPackage contains DependencyList which manually implements Deserialize, so I guess it's also necessary to manually implement Serialize.
Good chance to learn some serde.

[liranringel]

What would be the best way to calculate the checksum?
I thought to use cargo::ops::package to get the .crate file.

Edit: I noticed that the hash changes on every cargo package command, perhaps a timestamp is inserted, so this is meaningless to get the checksum.

[withoutboats]

@carols10cents Do you have strong opinions on whether this should live as a cargo subcommand or a separate package we publish to crates.io? My only concern is that cargo could become one of those tools where you type cargo --help and you're drowned in dozens of niche subcommands & you can't find what you're looking for. Do you think its too much to ask people to cargo install this?

[carols10cents]

I guess I don't feel super strongly about whether this is part of cargo or external, but it should be in the rust-lang org and it's seeming like having a library with a public api function as well as a binary would be useful...

[withoutboats]

I definitely agree about both of those points.

[liranringel]

So where should it be?
To make it clear, I interpret the task as getting a .crate file and generating it's index metadata.
If it won't be in cargo, it will probably be in a crate that depends on cargo, or cargo will depend on it (large refactor).

[cswindle]

Is anyone actively working on this, if not then I may make a start on a new crate to add the interface, this way it makes it a lot easier for versioning the interface for external crates to use (not to mention the reduction in time as cargo takes ages to compile when you are only needing a tiny amount of the function).

[liranringel]

I'd like to but no one answered me here or in gitter, and I don't want to make such important decisions by myself.

[cswindle]

After taking a look it would seem to me more useful to have a command which adds the entry in a git registry directly, for example this code. It just seems to me like anyone who uses a command to generate the fragment of index would need to also perform the following:

• update the git repo for the index
• calculate the index file location for the crate
• open the index file and append the new data to it
• commit the changes to the git repo
• push the changes to the git repo (I guess this could be optional for local git repos for testing)

That seems like a bunch of code which has already been written that anyone who uses the new command will also need to write (or copy). Is there some use case which I am missing?

[withoutboats]

We don't want it to do things that are highly contextual to specific users. Possibly it could make a git commit for you, but definitely not pull or push, since those involve questions of which remote is the source of truth, and what authentication do you use.

[cswindle]

Leaving out the pull and push and just updating a git repo would still make for quite a clean split, rather than spreading the knowledge of the registry layout around different crates.

[synek317]

I have implemented a working prototype here: https://github.com/synek317/cargo/tree/synek317_generate_index_metadata

I decided to create generate-index-metadata subcommand despite I'm not sure if it should be included in cargo. Anyway, this subcommand is just a 'working stub' (poor error message, almost no parameters).

[hallfox]

Any updates on this?

[synek317]

@hallfox have you checked my branch? I think it is ready but unfortunately I had no time to push it.

[ninrod]

with this issue resolved, would we entreprise users, trying to use rust behind a firewall, with access to an artifactory server, be finally able to use rust? we have no access whatsoever to crates.io.

[hallfox]

Yeah, I have seen it, was waiting for when you'd open a PR! Do you feel it's ready yet, or is there some blocker?

…

On Thu, Jun 21, 2018, 2:02 AM synek317 ***@***.***> wrote: @hallfox <https://github.com/hallfox> have you checked my branch? I think it is ready but unfortunately I had no time to push it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4497 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEfkSyl-BaOBPYfbbq-fpHbpiirAGHNTks5t-zbXgaJpZM4PZeP-> .

[synek317]

Actually, I was blocked with two things (+ time):

1. I'm not sure if this command should really be part of cargo (maybe cargo should only export some public items and generate-index-metadata should be external command?)

2. In order to check if everything works fine, I was building a tool that allows user to create crate registry in any github repository. I was close to finish a draft, but there are still some small things to do: https://github.com/synek317/cargo-git-registry

[repnop]

@synek317 have you had time to continue with this?

[ehuss]

Rather than implementing this in-tree, I have posted an experimental subcommand which relies on cargo metadata instead. It is available at https://crates.io/crates/cargo-index (requires the latest 1.33 nightly). If anyone is interested in it, feel free to send feedback on its issue tracker or take any of the code from within. In general it's a fairly straightforward translation of the cargo metadata output into the JSON structure needed in the index.

[synek317]

@ehuss this is not a good solution. I mean, I was also concerned if generate-index-metadata should be included in cargo or in subcommand as you can see in my comment above. Anyway registry structs are currently defined in crates.io and in cargo, as carols10cents pointed it out in the first comment. Instead of unifying it, you are adding another copy.

I've also some personal objections because now someone's effort (mine or yours, probably mine) will be wasted. If anyone had answered my previous question, I could just move it to the subcommand.

Not even mentioning that your implementation is something different than @carols10cents was asking in the first comment.

[ehuss]

@synek317 I apologize for the lack of coordination. The code I posted is not intended to be a permanent solution. It was intended to explore what would be needed to support this functionality outside of Cargo, and to gauge the utility.

The Cargo team is actively discussing the possibility of reorganizing some of the internals to make them more easily accessible in isolation and in a more stable form. However, it is not entirely clear, yet, exactly how that should appear, or how useful it would be. Different projects have different needs, so it's not always obvious how it should look and who it would serve.

[gilescope]

Given time has gone by, is there some feedback from people using https://crates.io/crates/cargo-index ? I have to say the docs didn't happen to mention it or I would have tried it out months ago. Maybe cargo local-registry should mention it in the alternatives section on their readme?