docs.rs link is missing from search results

[jturner314]

For crates without a documentation key in their Cargo.toml, crates.io is supposed to generate a Documentation link to docs.rs. However, this Documentation link is sometimes missing on search results.

Steps to reproduce:

1. Open a fresh browser instance. (For example, I'm using chromium --user-data-dir="$(mktemp -d)" on Arch Linux.)
2. Navigate to https://crates.io/search?q=ndarray-npy
3. Observe that there is no Documentation link for the ndarray-npy crate. (It should be there, but it is missing. That's what this issue is about.)
4. Click on the ndarray-npy crate link (URL: https://crates.io/crates/ndarray-npy).
5. Click the "back" button on your browser to navigate back to the previous page.
6. Observe that the Documentation link has now appeared.

I'm observing this behavior for crates that do not include a documentation key in their Cargo.toml. Currently, examples of this include the ndarray-npy, py_literal, aho-corasick, unreachable, and void crates, among many others.

[jtgeibel]

I think this is a general bug (not just for crates that are missing a documentation key). I think similar behavior will be seen with the following fields as well:

• keywords
• categories
• homepage
• documentation

(I'm not sure if any of these other fields show up on the results page. This may be the only missing field that is observable.)

The search results code constructs the crate data by using the minimal_encodable function, and this defaults those fields to null in the JSON response. Once you click through to the full page for the crate, then a full response is received and cached for the crate. This is why the link shows up after navigating back to the results.

[carols10cents]

When we added auto-linking to docs.rs in this PR, we discussed adding this to the search results page as well. However, we make a request to docs.rs to see if the docs were built successfully before adding the link to the page. We concluded that making a large number of requests to docs.rs wouldn't be a great user experience.

Some other thoughts:

• This is even more of a concern since we just increased the default number of results per page from 10 to 50.
• Now that we have a background job framework, after publishing a crate, we could enqueue a job that polls docs.rs until getting a build status or hitting some timeout, then storing the URL in that version's documentation field if the build_status is true and stopping polling if the build_status is false. Oops, just checked-- documentation is still a per-crate field, not per-version, so for this to work correctly we'd have to move that field to the versions table instead. Or we could watch the feed of releases from docs.rs, which would limit the requests we make of docs.rs to a number independent of the number of crates published. I think this is likely the direction I'd prefer, but whew, this is a bigger change than I expected.
• Onur mentioned in that PR thinking about adding a way to ask docs.rs about the status of many crates at once; as far as i can tell that hasn't been added but there are building blocks that wouldn't make it too bad. I don't know if it's worth adding such a thing to docs.rs if we're going to be the only consumer of it. Waiting for that one request to return and then updating the page to have a bunch of doc links or not might also be a non-optimal user experience.

[withoutboats]

Hey, not to be a bother but I just wanted to mention that now that the auto linking to docs.rs has existed for a while, it has become very common for libraries to no longer include their own documentation key in Cargo.toml. This means it increasingly seems like most libraries don't have links to documentation on the search index.

Sucks that this is such a big change, but if someone were looking for something to hack on in crates.io it would be very appreciated!

[jyn514]

Onur mentioned in that PR thinking about adding a way to ask docs.rs about the status of many crates at once; as far as i can tell that hasn't been added but there are building blocks that wouldn't make it too bad. I don't know if it's worth adding such a thing to docs.rs if we're going to be the only consumer of it. Waiting for that one request to return and then updating the page to have a bunch of doc links or not might also be a non-optimal user experience.

I would be ok with adding an API for fetching many crates at once. I do think we should tailor the API for this use case a bit and only show the status, not the crate details, to make it cheaper. (Right now we show a bunch of metadata about the versions: https://docs.rs/crate/tokio/1.0.0/builds.json)

[jtgeibel]

In our team meeting we discussed that on the search results page it probably makes sense to just link to the crate on docs.rs without providing a version. Then docs.rs would redirect to the latest rendered version available.

This might open small windows of inconsistency, such that if a new version was just published then the search results might show version 0.4.1 but the link redirects to 0.4.0 because the new version hasn't been rendered yet. Or a brand new crate could show up in search results but have no rendered version at all on docs.rs. However, this tradeoff seems acceptable in order to consistently show a fallback link to docs.rs if no link is explicitly provided in Cargo.toml.

[jyn514]

@jtgeibel what if the build for the crate failed? Then crates.io will link to the docs, but they won't be there.

This will also mean you get newer versions that the one currently published, which could be very confusing if they're different major versions.

[jyn514]

This will also mean you get newer versions

Oh wait, I guess crates.io search only shows the latest version. Never mind then.

[eggyal]

Perhaps docs.rs can provide a hook by which crates.io gets notified whenever a new doc build is completed? Then crates.io can update its metadata for that crate without having to make requests of docs.rs.

[jyn514]

Heh, docs.rs was actually hoping to get a hook from crates.io at some point: rust-lang/docs.rs#14. I guess it wouldn't hurt to have both.

[RalfJung]

Related cargo issue: rust-lang/cargo#11777

I honestly thought that not setting documentation is 100% equivalent to setting it to the docs.rs page. This automatic is-the-build-done check sounds like it's actually better to not add a link? Or does the check still happen if I set the link manually?

I think adding the documentation link to the search results as-if it was set in the Cargo.toml would already be a good step forward, even without doing any kind of fancy status check.

[epage]

I agree with the idea of removing the check. I could see the publish process falling back to this when populating the database with a backfill (upto a specific date if docs.rs never had a backfill). Then the javascript could be removed.

For cargo-info, we are taking the approach of always showing the link.

[epage]

For the gap between publish and built, some options

• Keep the javascript but assume its published if its weeks old.
• docs.rs posts a "build in progress" page that links to the queue. This would also be a usability improvement as i always struggle to find the queue.

For build errors, docs.rs should post a page, saying it failed, linking to the error with a link to the previous successful build.

[syphar]

[...] upto a specific date if docs.rs never had a backfill [...]

I don't know if docs.rs ever had an actual backfill, but I'm currently working on some changes to docs.rs ( specifically rust-lang/docs.rs#1011, with some more things attached), which will enable us running the consistency check, which will take care of adding all missing releases.

• docs.rs posts a "build in progress" page that links to the queue. This would also be a usability improvement as i always struggle to find the queue.

This will be made possible after rust-lang/docs.rs#1011 is done, and I was planning something like this.

For build errors, docs.rs should post a page, saying it failed, linking to the error with a link to the previous successful build.

We're already doing this.

So generally the question will be, just for search results, if you rather link to /latest/ on docs.rs, where it might not be the same version as stated in the search results, or you link to the specific latest version on crates.io, which might not be built on docs.rs yet.

And, even when we show in-progress builds on docs.rs, there would be a delay between the publish on crates.io, and docs.rs picking up the new release.

[RalfJung]

So generally the question will be, just for search results, if you rather link to /latest/ on docs.rs, where it might not be the same version as stated in the search results, or you link to the specific latest version on crates.io, which might not be built on docs.rs yet.

I think it should link to the version number that is shown in the search results. Then if that version is still building docs.rs can show a message saying "this is still in the queue" and offer a link to an earlier version that has already been built.