[rustdoc-json] JSON no longer inlines

[GuillaumeGomez]

Fixes #98007.
Fixes #96161.
Fixes #83057.
Fixes #83720.

I took over #93518 and applied the comments and added more tests.

There was one thing missing (which is in the second commit): if a non-exported item was used in a public API but not reexported, it was still missing.

cc @CraftSpider @Urgau @Enselic

r? @notriddle

[rustbot]

rustdoc-json-types is a public (although nightly-only) API. If possible, consider changing src/librustdoc/json/conversions.rs; otherwise, make sure you bump the FORMAT_VERSION constant.

cc @CraftSpider, @aDotInTheVoid

[CraftSpider]

This looks great to me, thank you for picking it up!

[GuillaumeGomez]

Fixed the tidy issue.

[Enselic]

Thanks a lot for working on this! ❤️

When I ran cargo public-api regression tests with this change, I caught a seemingly big semantic change, because some tests started failing.

Consider this code in src/lib.rs (I am using a type but e.g. fn has the same problem, and I imagine any other item type.):

#![feature(no_core)]
#![no_core]
#![no_std]
mod stripped {
    pub type T = usize;
}
pub use stripped::T;

before this PR, the pub use did not emit any JSON item by itself. Instead, there were only two items in rustdoc JSON:

• The crate module ("kind": "module")
• The typedef ("kind": "typedef")

After this change, the pub use itself is also emitted. This runs contrary to rustdoc HTML output, where the pub use will not be present by default. There are now four items:

• The crate module ("kind": "module", "is_crate": true)
• The typedef ("kind": "typedef")
• The stripped module ("kind": "module", "is_stripped": true)
• The pub use ("kind": "import")

I don't think we want the pub use JSON item in the output by default? Because that is not what cargo doc HTML shows by default.

...on the other hand, what should we do instead? We do not inline any longer.

Silly idea maybe, but can't we inline JSON as a very last step? So that the output becomes the same as before, but internally we do not do any inlining, so that all ICEs are fixed.

In short, this PR breaks my tool, so if we could go a bit gently forward, I would appreciate it :)

[Enselic]

StrippedModule is mentioned, but does not exist any longer, needs to be updated for Module::is_stripped.

[GuillaumeGomez]

Oh indeed, good catch!

[GuillaumeGomez]

The original PR did it the same way, so I'm surprised it's become an issue in-between. Inlining after will very likely be quite tricky and bring back most of the previous issues (conflicting IDs).

Also, the JSON output cannot be the same as the HTML one. Let me take an example:

mod private_mod {
    pub struct Foo;
}

pub fn foo() -> Foo {}

With this code, the HTML will not create a link for Foo because, since it's a private item, it doesn't have a page for itself. However, with the JSON output, we need to also get Foo so that ID doesn't link to a nonexistent item. Both are somewhat the same but the details actually make them very different. The JSON output needs to be more complete because it has to provide all the information to all its items.

Does it make sense? 😅

EDIT: I just thought about something: the JSON output was already showing pub use x::*; reexports whereas they're not displayed in the HTML output. So we're kinda making it more coherent. 😆
EDIT2: because they're directly inlined in the HTML output (which was the root of our issues in the JSON output).

[Enselic]

I've further looked into this change from my side this morning and have no blocking concerns any longer. Feel free to go ahead with merging this (as if I have any authority over that 😅).

(After this has landed I will experiment with letting my tool inline the JSON itself to get back the old behaviour. Seems like that could work. If it doesn't, it is not the end of the world.)

Looking forward to seeing those ICEs going away! 💪

The original PR did it the same way, so I'm surprised it's become an issue in-between.

This is probably not a new issue, it's simply that I never ran my test suite on the old PR, because

• I didn't have a test suite back then
• It didn't seem finished enough to test against

[GuillaumeGomez]

I've further looked into this change from my side this morning and have no blocking concerns any longer. Feel free to go ahead with merging this (as if I have any authority over that 😅).

Not an "authority" but you're a user providing feedback, which is very important for us. If there is no more issue for you, then I think we can indeed go forward. Thanks a lot for your help!

(After this has landed I will experiment with letting my tool inline the JSON itself to get back the old behaviour. Seems like that could work. If it doesn't, it is not the end of the world.)

I guess you already thought about it but if the type is a reexport, why not linking all its items directly into the top level on your side? That shouldn't require too much code hopefully.

Since @notriddle approved it too, let's go. :)

@bors r+

[bors]

📌 Commit 451e2f1bbfce876113cf7565015b219eb7e3e8d4 has been approved by GuillaumeGomez

It is now in the queue for this repository.

[GuillaumeGomez]

@bors r=notriddle

[bors]

💡 This pull request was already approved, no need to approve it again.

• There's another pull request that is currently being tested, blocking this pull request: [stable] 1.62.1 release #99299

[bors]

📌 Commit 451e2f1bbfce876113cf7565015b219eb7e3e8d4 has been approved by notriddle

It is now in the queue for this repository.

[GuillaumeGomez]

Just realized that I forgot to apply the comments...

@bors r-

[GuillaumeGomez]

Let's go again!

@bors r=notriddle

[Enselic]

As I was working on doing the inlining myself in my tool, I think I bumped into an unintended regression from this change.

Step-by-step:

Paste the following in either bash or zsh:

# Create a library with a trait with one associated function
echo '
pub trait A {
    fn trait_a_fn();
}
' > a.rs

# Create another library that depends on the first library
echo '
pub use a::A;
' > b.rs

# Check with what toolchains rustdoc JSON is generated for `trait_a_fn`
for toolchain in nightly-2022-07-16 nightly-2022-07-17; do
    # Create a directory with output from each toolchain
    mkdir -p $toolchain

    # Compile the library
    rustc +$toolchain --edition=2021 \
            --crate-type rlib -o $toolchain/liba.rlib a.rs

    # Generate rustdoc JSON for both libraries
    rustdoc +$toolchain --edition=2021 \
            --extern a=$toolchain/liba.rlib \
            -Z unstable-options --output-format json \
            -o $toolchain b.rs

    # See if rustdoc JSON for `trait_a_fn` was generated
    grep --silent trait_a_fn $toolchain/b.json && echo "rustdoc JSON for 'trait_a_fn' was generated by '$toolchain'"
done

Expected output:

rustdoc JSON for 'trait_a_fn' was generated by 'nightly-2022-07-16'
rustdoc JSON for 'trait_a_fn' was generated by 'nightly-2022-07-17'

Actual output:

rustdoc JSON for 'trait_a_fn' was generated by 'nightly-2022-07-16'

Would you agree this looks like an unintentional regression? Can you reproduce? Without rustdoc JSON for trait_a_fn I am not able to do the inlining myself.

Let me know if you would like me to file a separate issue for this, and I will happily do it.

[GuillaumeGomez]

No, it seems to be exactly the point of this PR (but maybe I misunderstood your issue?).

[Enselic]

Sorry for being unclear. My impression was that we always generated one big chunk of rustdoc JSON for an entire set of crates. But given your response, I'm starting to think that rustdoc JSON always was intended to generate rustdoc JSON only for one crate at a time. But then, the boundaries are not clear to me. The rustdoc JSON for the small example above contains literally thousands of items that does not have "crate_id": 0,, i.e. that comes from another crate. Looks like they come from libstd or similar though. Which maybe makes sense.

If rustdoc JSON is only designed and intended to be built for for one crate at a time, that means no tool can inline items of imports across crate boundaries, since the IDs are not stable across different builds of rustdoc JSON. If that is how it is supposed to be, that's fine I guess. At least for now :)

[GuillaumeGomez]

If a foreign type is present in the public API, it has its "definition" in the JSON as well.

[Enselic]

Only partial definition, it seems? Because even though the trait a::A in the sample code above is used by crate b, the rustdoc JSON for crate b only includes an item for the trait a::A, but not for the associated function fn trait_a_fn(). The rustdoc JSON for the trait A looks like this:

"20:3:1569": {
    "id": "20:3:1569",
    "crate_id": 20,
    "name": "A",
    "span": null,
    "visibility": "public",
    "docs": null,
    "links": {},
    "attrs": [],
    "deprecation": null,
    "kind": "trait",
    "inner": {
        "is_auto": false,
        "is_unsafe": false,
        "items": [
            "20:4:38964"
        ],
        "generics": {
            "params": [],
            "where_predicates": []
        },
        "bounds": [],
        "implementations": []
    }
},

but there is no rustdoc JSON item for the associated function fn trait_a_fn(). It's ID is there: "20:4:38964". But it has no corresponding item in the JSON. It is missing.

However, I just noticed that if crate b starts using fn trait_a_fn(), then it ends up in the rustdoc JSON.

To be crystal clear: This do give me the expected output:

# Create a library with a trait with one associated function
echo '
pub trait A {
    fn trait_a_fn();
}
' > a.rs

# Create another library that depends on the first library
echo '
pub use a::A;
pub struct S;
impl A for S {
    fn trait_a_fn() {}
}
' > b.rs

# Check with what toolchains rustdoc JSON is generated for `trait_a_fn`
for toolchain in nightly-2022-07-16 nightly-2022-07-17; do
    # Create a directory with output from each toolchain
    mkdir -p $toolchain

    # Compile the library
    rustc +$toolchain --edition=2021 \
            --crate-type rlib -o $toolchain/liba.rlib a.rs

    # Generate rustdoc JSON for both libraries
    rustdoc +$toolchain --edition=2021 \
            --extern a=$toolchain/liba.rlib \
            -Z unstable-options --output-format json \
            -o $toolchain b.rs

    # See if rustdoc JSON for `trait_a_fn` was generated
    grep --silent trait_a_fn $toolchain/b.json && echo "rustdoc JSON for 'trait_a_fn' was generated by '$toolchain'"
done

It could probably be argued that we should not emit rustdoc JSON only partially even in the initial case, but suddenly the current behaviour seems like much less of a problem.

[GuillaumeGomez]

So foreign items in "imported" foreign items are not in the JSON. Not sure if a bug or not at this point... Seems logical to have direct foreigns but not foreigns of foreigns I suppose.

[Enselic]

Actually, I draw conclusions too quickly. There is an item generated for fn trait_a_fn() for the impl, but not for the trait. I think this can only be seen as the rustdoc JSON being incomplete. I will file an issue for this at some point in the future, unless someone beats me to it.

[Urgau]

This new behavior feels "weird" but isn't really; you don't see the definition of a trait without clicking on it in the HTML and if it's a external crate it jumps in that crate you are not staying in the first crate.

My impression was that we always generated one big chunk of rustdoc JSON for an entire set of crates.

This is currently (was ?) the case but I don't think that something that we want. I think @CraftSpider talked about it sometimes ago, I remember something about pruning unnecessary stuff to reduce the size of the JSON and also to avoid this kind of problem.