New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace hoedown with pull in rustdoc #40338

Merged
merged 14 commits into from Mar 29, 2017

Conversation

Projects
None yet
@GuillaumeGomez
Member

GuillaumeGomez commented Mar 8, 2017

cc @rust-lang/docs

@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Mar 8, 2017

r? @steveklabnik

(rust_highfive has picked a reviewer for you, use r? to override)

@steveklabnik

I will give this a real review tomorrow, but 🎊

@@ -179,8 +181,8 @@ extern {
fn hoedown_document_free(md: *mut hoedown_document);

fn hoedown_buffer_new(unit: libc::size_t) -> *mut hoedown_buffer;
fn hoedown_buffer_put(b: *mut hoedown_buffer, c: *const libc::c_char,
n: libc::size_t);
/*fn hoedown_buffer_put(b: *mut hoedown_buffer, c: *const libc::c_char,

This comment has been minimized.

@steveklabnik

steveklabnik Mar 8, 2017

Member

Why leave this in but commented out?

@@ -594,7 +598,7 @@ impl<'a> fmt::Display for MarkdownHtml<'a> {
}

pub fn plain_summary_line(md: &str) -> String {
extern fn link(_ob: *mut hoedown_buffer,
/*extern fn link(_ob: *mut hoedown_buffer,

This comment has been minimized.

@steveklabnik

steveklabnik Mar 8, 2017

Member

same here

let mut register_header = None;
'main: loop {
let next_event = parser.next();
if let Some(event) = next_event {

This comment has been minimized.

@frewsxcv

frewsxcv Mar 8, 2017

Member

If you made this:

let event = match next_event {
    Some(event) => event,
    None => break,
};
....

This would reduce the indentation

This comment has been minimized.

@frewsxcv

frewsxcv Mar 8, 2017

Member

At that point, couldn't this be a:

while let Some(event) = parser.next() {
    ...
}

@GuillaumeGomez GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch from 9268ec6 to 2c33d62 Mar 8, 2017

@steveklabnik

Seems good to me, though I agree with @frewsxcv 's comments about nesting

@GuillaumeGomez GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch 4 times, most recently from 376b5dd to 2537b41 Mar 8, 2017

@GuillaumeGomez GuillaumeGomez changed the title from [WIP] Replace hoedown with pull in rustdoc to Replace hoedown with pull in rustdoc Mar 10, 2017

@GuillaumeGomez

This comment has been minimized.

Member

GuillaumeGomez commented Mar 10, 2017

This is now ready to review!

let mut content = String::new();
loop {
let event = parser.next();
if let Some(event) = event {

This comment has been minimized.

@frewsxcv

frewsxcv Mar 10, 2017

Member

Could this could just be a for loop?

This comment has been minimized.

@killercup

killercup Mar 10, 2017

Member

Agreed, this is for event in parser { match event { … } } (see main review comment)

fn link(parser: &mut Parser, buffer: &mut String, url: &str, mut title: String) {
loop {
let event = parser.next();
if let Some(event) = event {

This comment has been minimized.

@frewsxcv

frewsxcv Mar 10, 2017

Member

for loop?

let mut content = String::new();
loop {
let event = parser.next();
if let Some(event) = event {

This comment has been minimized.

@frewsxcv

frewsxcv Mar 10, 2017

Member

for loop?

use std::fmt::{self, Write};
use std::slice;
//use std::slice;

This comment has been minimized.

@frewsxcv

frewsxcv Mar 10, 2017

Member

Remove commented out imports

/// A unit struct which has the `fmt::Display` trait implemented. When
/// formatted, this struct will emit the HTML corresponding to the rendered
/// version of the contained markdown string.
pub struct Markdown<'a>(pub &'a str);
// The second parameter is whether we need a shorter version or not.

This comment has been minimized.

@frewsxcv

frewsxcv Mar 10, 2017

Member

What is a "shorter version"?

Maybe this should be a struct with named fields instead of using unnamed fields

This comment has been minimized.

@steveklabnik

steveklabnik Mar 10, 2017

Member

yeah, boolean parameters are usually a code smell

This comment has been minimized.

@killercup

killercup Mar 10, 2017

Member

I think I'd prefer an enum MarkdownOutputStyle { Compact, Fancy } (or some other fitting variant names) instead of bool

This comment has been minimized.

@GuillaumeGomez
@@ -26,13 +26,13 @@

#![allow(non_camel_case_types)]

use libc;
//use libc;

This comment has been minimized.

@frewsxcv

frewsxcv Mar 10, 2017

Member

Do we still use the libc crate in rustdoc?

(b'0' <= c && c <= b'9') ||
c == b'-' || c == b'_' || c == b'.' ||
c == b'~' || c == b'!' || c == b'\'' ||
c == b'(' || c == b')' || c == b'*'

This comment has been minimized.

@frewsxcv

frewsxcv Mar 10, 2017

Member

Pull in rust-url for this?

This comment has been minimized.

@GuillaumeGomez

GuillaumeGomez Mar 10, 2017

Member

This is not part of the change, however I can do it next up.

}

fn looper<'a>(parser: &'a mut Parser, buffer: &mut String, next_event: Option<Event<'a>>,
toc_builder: &mut Option<TocBuilder>, shorter: bool) -> bool {
if let Some(event) = next_event {

This comment has been minimized.

@frewsxcv

frewsxcv Mar 10, 2017

Member

Could return early here if this is None

This comment has been minimized.

@GuillaumeGomez

GuillaumeGomez Mar 10, 2017

Member

Doesn't change much from my point of view... I think this syntax is actually better.

@steveklabnik

This comment has been minimized.

Member

steveklabnik commented Mar 10, 2017

/cc @raphlinus ❤️

@steveklabnik

This comment has been minimized.

Member

steveklabnik commented Mar 10, 2017

So, I'm spot-checking some text to find regressions.

std::collections module page

collections

std::convert module page

convert

prelude:

prelude

some smooshed text

std::sync::LockResult

lock

std::string

string

that's what i found just now. we may need to tweak some text, and it's probably good to go through the whole entire thing once the technical side is good and clear up these kinds of issues.

@killercup

Nice to see some progress here!

One pattern I immediately saw was all the loop constructs matching the event and tag. I'd:

  1. use Event::* and use Tag::*; to reduce the visual noise

  2. Try to rewrite them as let events = parser.$filter_relevant_events; for event in events { match event {…} }, where $filter_relevant_events are some operations on the iterator to get the events you are interested in (e.g., to use take_while instead of adding break arms).

    FYI, I wrote a macro once for this exact use case to allow events.take_while(not!(end Tag::Paragraph)) and similar. This may help converting most of these loops into nice iterators.

    You can find it and example using it here. You'll probably need to extend it to support multiple end tag variants as used below.

I'd also really try to get rid of the shorter: bool stuff and replace with some semantically meaningful names :)

Switching parser is not something I expect to just work, even when both apparently conform to the CommonMark spec. Before merging this, I'd render a bunch of docs (std and some larger crates), and try to

  • set up a script that opens and takes screenshots of all generated html pages to visually diff them with ones generated by the current stable rustdoc,
  • or let humans look at the rendered pages,
  • or do both 😄
@@ -1203,6 +1203,7 @@ dependencies = [
"build_helper 0.1.0",
"gcc 0.3.43 (registry+https://github.com/rust-lang/crates.io-index)",
"log 0.0.0",
"pulldown-cmark 0.0.8 (registry+https://github.com/rust-lang/crates.io-index)",

This comment has been minimized.

@killercup

killercup Mar 10, 2017

Member

0.0.8… Another lib we should help get to 1.0

}
}
buffer.push_str(&format!("<table>{}{}</table>",
content,
if shorter || rows.is_empty() {

This comment has been minimized.

@killercup

killercup Mar 10, 2017

Member

Is this the only place where shorter is actually used? The name shorter does not imply "no tbody tag" to me…

This comment has been minimized.

@GuillaumeGomez

GuillaumeGomez Mar 10, 2017

Member

In paragraph as well. The point is to display only the first line, so everything that could be on more than one line use it.

/// A unit struct which has the `fmt::Display` trait implemented. When
/// formatted, this struct will emit the HTML corresponding to the rendered
/// version of the contained markdown string.
pub struct Markdown<'a>(pub &'a str);
// The second parameter is whether we need a shorter version or not.

This comment has been minimized.

@killercup

killercup Mar 10, 2017

Member

I think I'd prefer an enum MarkdownOutputStyle { Compact, Fancy } (or some other fitting variant names) instead of bool

let mut content = String::new();
loop {
let event = parser.next();
if let Some(event) = event {

This comment has been minimized.

@killercup

killercup Mar 10, 2017

Member

Agreed, this is for event in parser { match event { … } } (see main review comment)

_ => {}
}
} else {
break 'main;

This comment has been minimized.

@killercup

killercup Mar 10, 2017

Member

finally, a good reason not use a for loop ;)

@GuillaumeGomez

This comment has been minimized.

Member

GuillaumeGomez commented Mar 11, 2017

Most problems should be solved now. I'll write a macro next to allow to remove all these loops.

@ollie27

This comment has been minimized.

Contributor

ollie27 commented Mar 11, 2017

Is there any reason you're not using pulldown_cmark::html::push_html to actually render the HTML?

@bors

This comment has been minimized.

Contributor

bors commented Mar 11, 2017

☔️ The latest upstream changes (presumably #40432) made this pull request unmergeable. Please resolve the merge conflicts.

@GuillaumeGomez

This comment has been minimized.

Member

GuillaumeGomez commented Mar 11, 2017

@ollie27: Unfortunately yes: the generation of urls mainly (for play.rust-lang in blockcodes).

"&lt;", "&gt;", "&amp;", "&#39;", "&quot;"];
for sub in repl_sub {
id = id.replace(sub, "");
}

This comment has been minimized.

@steveklabnik

steveklabnik Mar 11, 2017

Member

shouldn't this mean the comment above goes away too?

This comment has been minimized.

@GuillaumeGomez

GuillaumeGomez Mar 11, 2017

Member

Absolutely!

@GuillaumeGomez GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch 2 times, most recently from c132361 to 8b61716 Mar 11, 2017

@killercup

Code looks nicer with the macro, though I'd probably make it more generic.

Also, why not use Event::* and use Tag::*?

@@ -104,6 +104,25 @@ thread_local!(pub static PLAYGROUND: RefCell<Option<(Option<String>, String)>> =
RefCell::new(None)
});

macro_rules! event_loop_break {

This comment has been minimized.

@killercup

killercup Mar 11, 2017

Member

Nice. I'd call it collect_md_events, though, and make calling it a bit more expressive, e.g.

collect_md_events!(from parser into &mut buf (toc_builder, shorter) until End(Header(_)))

This comment has been minimized.

@GuillaumeGomez

GuillaumeGomez Mar 12, 2017

Member

Not sure if this is super useful. Anyone else has an opinion on it?

shorter: MarkdownOutputStyle) -> fmt::Result {
fn block(parser: &mut Parser, buffer: &mut String, lang: &str) {
let mut origtext = String::new();
while let Some(event) = parser.next() {

This comment has been minimized.

@killercup

killercup Mar 11, 2017

Member

Could also be using the macro (e.g. when calling without toc builder and shorter)

This comment has been minimized.

@GuillaumeGomez

GuillaumeGomez Mar 12, 2017

Member

I didn't see much interest in expanding the macro to handle this case. But yes it's possible.

@GuillaumeGomez GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch from 3c6a31f to 7f190fe Mar 12, 2017

@GuillaumeGomez

This comment has been minimized.

Member

GuillaumeGomez commented Mar 12, 2017

Also, why not use Event::* and use Tag::*?

Simply because I like the code being explicit. At least with this syntax, you don't have to think about it, the information is already there.

@ollie27

This comment has been minimized.

Contributor

ollie27 commented Mar 12, 2017

@ollie27: Unfortunately yes: the generation of urls mainly (for play.rust-lang in blockcodes).

You can still use push_html and add these links by implementing an iterator adapter around Parser. It would pass most Events through but you could filter and modify the ones you're interested in like CodeBlock and Header. This way you don't have to reimplement all the HTML rendering that pulldown-cmark already provides.

@GuillaumeGomez GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch from 83972b3 to a7c6d3e Mar 28, 2017

@GuillaumeGomez

This comment has been minimized.

Member

GuillaumeGomez commented Mar 28, 2017

Updated.

@frewsxcv

I have a bunch of nitpicky style things, but they aren't important enough to hold up this PR any longer. Looks good to me!

@GuillaumeGomez

This comment has been minimized.

Member

GuillaumeGomez commented Mar 29, 2017

@frewsxcv, @steveklabnik: Since both of you agreed on the changes, I'll r+ this PR. Don't hesitate to r- if you find anything!

@bors: r=frewsxcv,steveklabnik

@bors

This comment has been minimized.

Contributor

bors commented Mar 29, 2017

📌 Commit a7c6d3e has been approved by frewsxcv,steveklabnik

@bors

This comment has been minimized.

Contributor

bors commented Mar 29, 2017

⌛️ Testing commit a7c6d3e with merge abf5592...

bors added a commit that referenced this pull request Mar 29, 2017

Auto merge of #40338 - GuillaumeGomez:pulldown-switch, r=frewsxcv,ste…
…veklabnik

Replace hoedown with pull in rustdoc

cc @rust-lang/docs
@bors

This comment has been minimized.

Contributor

bors commented Mar 29, 2017

☀️ Test successful - status-appveyor, status-travis
Approved by: frewsxcv,steveklabnik
Pushing abf5592 to master...

@bors bors merged commit a7c6d3e into rust-lang:master Mar 29, 2017

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
homu Test successful
Details
@ollie27

This comment has been minimized.

Contributor

ollie27 commented Mar 29, 2017

Was this actually finished? It's missing support for horizontal rules, footnotes and images. There are also several bugs like missing the <br /> for HardBreaks, the title part of links being rendered as visible text rather than as an attribute on the <a> tag and MarkdownHtml now doing the same as Markdown when it should be escaping raw HTML.

As I previously mentioned, using push_html would have been much easier than trying to reimplement all of the HTML rendering.

@frewsxcv

This comment has been minimized.

Member

frewsxcv commented Mar 29, 2017

Was this actually finished? It's missing support for horizontal rules, footnotes and images. There are also several bugs like missing the <br /> for HardBreaks, the title part of links being rendered as visible text rather than as an attribute on the <a> tag and MarkdownHtml now doing the same as Markdown when it should be escaping raw HTML.

Have you confirmed any of these? I can bring up these concerns during today's Documentation Team meeting.

@ollie27

This comment has been minimized.

Contributor

ollie27 commented Mar 29, 2017

Have you confirmed any of these?

Yes. Try rendering the following in the current rustdoc and after this change:

/// markdown test
///
/// this is a [link].
///
/// [link]: https://example.com "this is a title"
///
/// hard break:  
/// after hard break
///
/// -----------
///
/// a footnote[^footnote].
///
/// [^footnote]: Thing
///
/// ![Rust](https://www.rust-lang.org/logos/rust-logo-128x128-blk-v2.png)
#[deprecated(note = "Struct<T>")]
pub fn f() {}

Before:
Before

After:
After

@steveklabnik

This comment has been minimized.

Member

steveklabnik commented Mar 29, 2017

Two things:

  1. this landed after last night's nightly, so tonight's is the first night where it lands. I was already planning on trying to publicize "please look at this since it's a big change", so knowing this stuff is good.
  2. I am currently building docs with the commit before and the commit after, and then i'm going to diff them to see exactly what's different.

I don't think that this missing a few things is the end of the world; now that this has landed, fixing stuff up is much easier. We've still got plenty of time before the release; four weeks until beta, and then we could backport anything catastrophic. Getting this out in the wild sooner rather than later is important IMHO.

@GuillaumeGomez GuillaumeGomez deleted the GuillaumeGomez:pulldown-switch branch Mar 29, 2017

@steveklabnik

This comment has been minimized.

Member

steveklabnik commented Mar 29, 2017

Here is an un-semantic diff steveklabnik/docdiff@5d17061

@steveklabnik

This comment has been minimized.

Member

steveklabnik commented Mar 29, 2017

@steveklabnik steveklabnik referenced this pull request Mar 29, 2017

Closed

Tracking issue for hoedown -> pulldown regressions #40912

13 of 13 tasks complete
@steveklabnik

This comment has been minimized.

Member

steveklabnik commented Mar 29, 2017

I've filed #40912 to track these regressions. 😄

@mbrubeck

This comment has been minimized.

Contributor

mbrubeck commented Sep 28, 2017

This was disabled by default in #41431, and #44229 is tracking the plan to enable it by default.

@nrc nrc referenced this pull request Sep 28, 2017

Closed

Tracking issue (Rustdoc): hoedown -> pulldown migration #44229

10 of 10 tasks complete

GuillaumeGomez added a commit to GuillaumeGomez/rust that referenced this pull request Feb 16, 2018

Remove hoedown from rustdoc
Is it really time? Have our months, no, *years* of suffering come to an end? Are we finally able to cast off the pall of Hoedown? The weight which has dragged us down for so long?

-----

So, timeline for those who need to catch up:

* Way back in December 2016, [we decided we wanted to switch out the markdown renderer](rust-lang#38400). However, this was put on hold because the build system at the time made it difficult to pull in dependencies from crates.io.
* A few months later, in March 2017, [the first PR was done, to switch out the renderers entirely](rust-lang#40338). The PR itself was fraught with CI and build system issues, but eventually landed.
* However, not all was well in the Rustdoc world. During the PR and shortly after, we noticed [some differences in the way the two parsers handled some things](rust-lang#40912), and some of these differences were major enough to break the docs for some crates.
* A couple weeks afterward, [Hoedown was put back in](rust-lang#41290), at this point just to catch tests that Pulldown was "spuriously" running. This would at least provide some warning about spurious tests, rather than just breaking spontaneously.
* However, the problems had created enough noise by this point that just a few days after that, [Hoedown was switched back to the default](rust-lang#41431) while we came up with a solution for properly warning about the differences.
* That solution came a few weeks later, [as a series of warnings when the HTML emitted by the two parsers was semantically different](rust-lang#41991). But that came at a cost, as now rustdoc needed proc-macro support (the new crate needed some custom derives farther down its dependency tree), and the build system was not equipped to handle it at the time. It was worked on for three months as the issue stumped more and more people.
  * In that time, [bootstrap was completely reworked](rust-lang#43059) to change how it ordered compilation, and [the method by which it built rustdoc would change](rust-lang#43482), as well. This allowed it to only be built after stage1, when proc-macros would be available, allowing the "rendering differences" PR to finally land.
  * The warnings were not perfect, and revealed a few [spurious](rust-lang#44368) [differences](rust-lang#45421) between how we handled the renderers.
  * Once these were handled, [we flipped the switch to turn on the "rendering difference" warnings all the time](rust-lang#45324), in October 2017. This began the "warning cycle" for this change, and landed in stable in 1.23, on 2018-01-04.
  * Once those warnings hit stable, and after a couple weeks of seeing whether we would get any more reports than what we got from sitting on nightly/beta, [we switched the renderers](rust-lang#47398), making Pulldown the default but still offering the option to use Hoedown.

And that brings us to the present. We haven't received more new issues from this in the meantime, and the "switch by default" is now on beta. Our reasoning is that, at this point, anyone who would have been affected by this has run into it already.

GuillaumeGomez added a commit to GuillaumeGomez/rust that referenced this pull request Feb 16, 2018

Remove hoedown from rustdoc
Is it really time? Have our months, no, *years* of suffering come to an end? Are we finally able to cast off the pall of Hoedown? The weight which has dragged us down for so long?

-----

So, timeline for those who need to catch up:

* Way back in December 2016, [we decided we wanted to switch out the markdown renderer](rust-lang#38400). However, this was put on hold because the build system at the time made it difficult to pull in dependencies from crates.io.
* A few months later, in March 2017, [the first PR was done, to switch out the renderers entirely](rust-lang#40338). The PR itself was fraught with CI and build system issues, but eventually landed.
* However, not all was well in the Rustdoc world. During the PR and shortly after, we noticed [some differences in the way the two parsers handled some things](rust-lang#40912), and some of these differences were major enough to break the docs for some crates.
* A couple weeks afterward, [Hoedown was put back in](rust-lang#41290), at this point just to catch tests that Pulldown was "spuriously" running. This would at least provide some warning about spurious tests, rather than just breaking spontaneously.
* However, the problems had created enough noise by this point that just a few days after that, [Hoedown was switched back to the default](rust-lang#41431) while we came up with a solution for properly warning about the differences.
* That solution came a few weeks later, [as a series of warnings when the HTML emitted by the two parsers was semantically different](rust-lang#41991). But that came at a cost, as now rustdoc needed proc-macro support (the new crate needed some custom derives farther down its dependency tree), and the build system was not equipped to handle it at the time. It was worked on for three months as the issue stumped more and more people.
  * In that time, [bootstrap was completely reworked](rust-lang#43059) to change how it ordered compilation, and [the method by which it built rustdoc would change](rust-lang#43482), as well. This allowed it to only be built after stage1, when proc-macros would be available, allowing the "rendering differences" PR to finally land.
  * The warnings were not perfect, and revealed a few [spurious](rust-lang#44368) [differences](rust-lang#45421) between how we handled the renderers.
  * Once these were handled, [we flipped the switch to turn on the "rendering difference" warnings all the time](rust-lang#45324), in October 2017. This began the "warning cycle" for this change, and landed in stable in 1.23, on 2018-01-04.
  * Once those warnings hit stable, and after a couple weeks of seeing whether we would get any more reports than what we got from sitting on nightly/beta, [we switched the renderers](rust-lang#47398), making Pulldown the default but still offering the option to use Hoedown.

And that brings us to the present. We haven't received more new issues from this in the meantime, and the "switch by default" is now on beta. Our reasoning is that, at this point, anyone who would have been affected by this has run into it already.

GuillaumeGomez added a commit to GuillaumeGomez/rust that referenced this pull request Feb 16, 2018

Remove hoedown from rustdoc
Is it really time? Have our months, no, *years* of suffering come to an end? Are we finally able to cast off the pall of Hoedown? The weight which has dragged us down for so long?

-----

So, timeline for those who need to catch up:

* Way back in December 2016, [we decided we wanted to switch out the markdown renderer](rust-lang#38400). However, this was put on hold because the build system at the time made it difficult to pull in dependencies from crates.io.
* A few months later, in March 2017, [the first PR was done, to switch out the renderers entirely](rust-lang#40338). The PR itself was fraught with CI and build system issues, but eventually landed.
* However, not all was well in the Rustdoc world. During the PR and shortly after, we noticed [some differences in the way the two parsers handled some things](rust-lang#40912), and some of these differences were major enough to break the docs for some crates.
* A couple weeks afterward, [Hoedown was put back in](rust-lang#41290), at this point just to catch tests that Pulldown was "spuriously" running. This would at least provide some warning about spurious tests, rather than just breaking spontaneously.
* However, the problems had created enough noise by this point that just a few days after that, [Hoedown was switched back to the default](rust-lang#41431) while we came up with a solution for properly warning about the differences.
* That solution came a few weeks later, [as a series of warnings when the HTML emitted by the two parsers was semantically different](rust-lang#41991). But that came at a cost, as now rustdoc needed proc-macro support (the new crate needed some custom derives farther down its dependency tree), and the build system was not equipped to handle it at the time. It was worked on for three months as the issue stumped more and more people.
  * In that time, [bootstrap was completely reworked](rust-lang#43059) to change how it ordered compilation, and [the method by which it built rustdoc would change](rust-lang#43482), as well. This allowed it to only be built after stage1, when proc-macros would be available, allowing the "rendering differences" PR to finally land.
  * The warnings were not perfect, and revealed a few [spurious](rust-lang#44368) [differences](rust-lang#45421) between how we handled the renderers.
  * Once these were handled, [we flipped the switch to turn on the "rendering difference" warnings all the time](rust-lang#45324), in October 2017. This began the "warning cycle" for this change, and landed in stable in 1.23, on 2018-01-04.
  * Once those warnings hit stable, and after a couple weeks of seeing whether we would get any more reports than what we got from sitting on nightly/beta, [we switched the renderers](rust-lang#47398), making Pulldown the default but still offering the option to use Hoedown.

And that brings us to the present. We haven't received more new issues from this in the meantime, and the "switch by default" is now on beta. Our reasoning is that, at this point, anyone who would have been affected by this has run into it already.

GuillaumeGomez added a commit to GuillaumeGomez/rust that referenced this pull request Feb 16, 2018

Remove hoedown from rustdoc
Is it really time? Have our months, no, *years* of suffering come to an end? Are we finally able to cast off the pall of Hoedown? The weight which has dragged us down for so long?

-----

So, timeline for those who need to catch up:

* Way back in December 2016, [we decided we wanted to switch out the markdown renderer](rust-lang#38400). However, this was put on hold because the build system at the time made it difficult to pull in dependencies from crates.io.
* A few months later, in March 2017, [the first PR was done, to switch out the renderers entirely](rust-lang#40338). The PR itself was fraught with CI and build system issues, but eventually landed.
* However, not all was well in the Rustdoc world. During the PR and shortly after, we noticed [some differences in the way the two parsers handled some things](rust-lang#40912), and some of these differences were major enough to break the docs for some crates.
* A couple weeks afterward, [Hoedown was put back in](rust-lang#41290), at this point just to catch tests that Pulldown was "spuriously" running. This would at least provide some warning about spurious tests, rather than just breaking spontaneously.
* However, the problems had created enough noise by this point that just a few days after that, [Hoedown was switched back to the default](rust-lang#41431) while we came up with a solution for properly warning about the differences.
* That solution came a few weeks later, [as a series of warnings when the HTML emitted by the two parsers was semantically different](rust-lang#41991). But that came at a cost, as now rustdoc needed proc-macro support (the new crate needed some custom derives farther down its dependency tree), and the build system was not equipped to handle it at the time. It was worked on for three months as the issue stumped more and more people.
  * In that time, [bootstrap was completely reworked](rust-lang#43059) to change how it ordered compilation, and [the method by which it built rustdoc would change](rust-lang#43482), as well. This allowed it to only be built after stage1, when proc-macros would be available, allowing the "rendering differences" PR to finally land.
  * The warnings were not perfect, and revealed a few [spurious](rust-lang#44368) [differences](rust-lang#45421) between how we handled the renderers.
  * Once these were handled, [we flipped the switch to turn on the "rendering difference" warnings all the time](rust-lang#45324), in October 2017. This began the "warning cycle" for this change, and landed in stable in 1.23, on 2018-01-04.
  * Once those warnings hit stable, and after a couple weeks of seeing whether we would get any more reports than what we got from sitting on nightly/beta, [we switched the renderers](rust-lang#47398), making Pulldown the default but still offering the option to use Hoedown.

And that brings us to the present. We haven't received more new issues from this in the meantime, and the "switch by default" is now on beta. Our reasoning is that, at this point, anyone who would have been affected by this has run into it already.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment