Add support to display documentation for one crate as one single page to docs.rs

[magnusart]

I'm using cursor.sh and it would be very useful to be able to index a full docs.rs crate so that the AI-assistant can use that.

An example to illustrate could be https://expressjs.com/en/5x/api.html. One page with all documenatation for the specific crate. That makes it much easier for tools to crawl and create local embeddings rather than follow a links to see Functions, Traits, Structs, Macros etc.

[Nemo157]

This would require a rustdoc feature, docs.rs just serves the pages it generates.

[Nemo157]

(There is a single-file archive available, https://docs.rs/about/download, so you might be able to grab that and just concatenate all the pages together).

[syphar]

On top of what Nemo said, depending on the size of the docs this might need quite some special handling. The most extreme cases we have contain multiple GB of docs (compressed!) and several million files.

[syphar]

I wonder if the AI assistant could just use the rustdoc JSON output, which already contains all in one file. Though many relations are probably not clearly visible in there?

[magnusart]

Thanks for the answers. @syphar is that JSON output available online or would I need to generate it myself?

[syphar]

( badly formatted because I'm on my phone)

We want to build and host the output at some point: #1285

But since the format is not stable yet, we're reluctant adding it.

So currently you would have to run it yourself locally, as some crates already do (cargo-semver-check is one example)