Add syntax highlighting for code blocks (JSON, etc.)

[fasterthanlime]

...starting with JSON, which is surprisingly monocolor right now.

jyn says it's hard, I don't yet know it's hard, where would one got to help with this?

[jyn514]

i’ll quote my comments:

this is stupidly hard because currently the syntax highlighting comes directly from rustdoc. it might be possible to add something in the html rewriter, that happens lazily when people make a request instead of when the crate is built. but it’s, like, not trivial

Amos said in response

Yeah, I wouldn't want it to be on the fly. I would like it to be part of the build.

which i think is just not feasible.

[syphar]

Is this about syntax highlighting in our own source code views?

or in the rustdoc source views?

[syphar]

There is also #2577 which is roughly what we talk about? If it's about code blocks in docs?

[GuillaumeGomez]

Quoting my answer from mastodon:

In short, the preferred way would be to add highlight.js and people would add it into their Cargo.toml somthing like extra-js = ["highlight.js"]

Although we're currently in the middle of a migration so it will likely wait a bit more.

[syphar]

Current I'm not sure about the implications about adding custom JS libs, and maintaining support for them forever.

Isn't this something that all crates might want? not only selective ones?

[fasterthanlime]

In short, the preferred way would be to add highlight.js and people would add it into their Cargo.toml somthing like extra-js = ["highlight.js"]

Although we're currently in the middle of a migration so it will likely wait a bit more.

I would like to preface this by saying that I am obviously not a docs.rs maintainer and I will quickly fold / bow to whatever the opinion of the actual maintainers are. But I would like to understand more about why it's so infeasible to add syntax highlighting as part of the build in rustdoc itself.

Is it because it would make the binary too large? Is it because it would expand the surface of what has to be maintained by the core team beyond their resources and or will to live? Is it for a secret third reason?

I'm just trying to understand why the answer is no.

And then I'll be happy and just, yeah, consider something client-side, I guess.

[GuillaumeGomez]

Isn't this something that all crates might want? not only selective ones?

I don't think it's that common for crates to have code blocks from other languages. However we could detect it (I mean, we generate JSON output too now =D) and add the JS in these cases.

[jyn514]

How is Rust code highlighted then?

@fasterthanlime the model of docs.rs is something like this:

• on each crates.io publish, docs.rs pulls the git repo for the crates.io index, computes a diff, and queues all crates that were published since the last pull.
• there is a “builder” continuously running (i think several concurrently since recently). the builder is essentially running cargo doc with sandboxing, it’s more complicated but not in ways you care about. we have the ability to pass rustdoc custom flags, and to modify rustdoc upstream, but we do not want to have a fork of rustdoc. this is the point at which rustdoc adds syntax highlighting.
• once docs are built, we upload them to S3 where they stay indefinitely. they are never modified once built. last i checked in 2022 there were several terabytes of docs.
• when docs are requested, we fetch them from S3 and rewrite them to insert the docs.rs header.

The problem with changing this at build time is that a) it doesn’t help the several terabytes of existing docs, and b) it adds a bunch of upstream maintenance to rustdoc. It’s much harder for rustdoc to add and remove things than for docs.rs to; and rustdoc has to be very careful about what JS it uses.

[GuillaumeGomez]

Is it because it would make the binary too large? Is it because it would expand the surface of what has to be maintained by the core team beyond their resources and or will to live? Is it for a secret third reason?

This is just my opinion on this, not an official position of the rustdoc team. For me, rustdoc should only care about rust. We added options to allow people to customize their doc builds in case they wanted more so we (rustdoc team) wouldn't need to maintain things like syntax highlighting for other languages, or graphs, or latex, etc. I still think it's what's best considering that we are a very limited number of contributors and even less maintainers.

PS: @jyn514 wrote a very nice (and more complete) sum up. :)

[fasterthanlime]

We added options to allow people to customize their doc builds in case they wanted more so we (rustdoc team) wouldn't need to maintain things like syntax highlighting for other languages, or graphs, or latex, etc. I still think it's what's best considering that we are a very limited number of contributors and even less maintainers.

That sounds great and super reasonable! Can we have customization in the form of WASM modules that are run during the build instead of JS bundles that are run on people's browsers? 😅

Semi-jokes aside I totally understand and I figured as much. I just want to point out that tree-sitter makes both parsing and highlighting a solved problem (today! wasn't the case even a few years ago). And, you know, apart from the binary size, which is an actual issue, the maintenance burden is really, really low.

Wait, are you saying if I join the team and I pinky swear not to disappear and to take care of that, then it might happen?

[GuillaumeGomez]

Semi-jokes aside I totally understand and I figured as much. I just want to point out that tree-sitter makes both parsing and highlighting a solved problem (today! wasn't the case even a few years ago). And, you know, apart from the binary size, which is an actual issue, the maintenance burden is really, really low.

Still more than nothing to maintain, and would generate much bigger HTML (which is becoming an issue we don't really know how to solve currently ^^').

Wait, are you saying if I join the team and I pinky swear not to disappear and to take care of that, then it might happen?

But first you need to have an RFC, then implement it, write a lot of tests... You're gonna have a lot of fun, can't wait. 😈

[jyn514]

I just want to point out that tree-sitter makes both parsing and highlighting a solved problem

i have exactly zero confidence this is true on windows. dlopen is a horrible plugin mechanism.

i wrote up more about tree sitter here: https://codeberg.org/jyn514/flower/issues/22#issuecomment-6801112

in general @fasterthanlime I think you are used to working in your personal monorepo. rustdoc has to work in a much wider variety of much more hostile conditions. finding something that works in all those places is much harder than something that works in only one environment.