Replace hoedown with pull in rustdoc #40338

GuillaumeGomez · Mar 8, 2017

cc @rust-lang/docs

rust-highfive · Mar 8, 2017

r? @steveklabnik

(rust_highfive has picked a reviewer for you, use r? to override)

steveklabnik · Mar 8, 2017

steveklabnik reviewed Mar 8, 2017

View changes

I will give this a real review tomorrow, but 🎊

steveklabnik · Mar 8, 2017

src/librustdoc/html/markdown.rs

@@ -179,8 +181,8 @@ extern {
    fn hoedown_document_free(md: *mut hoedown_document);

    fn hoedown_buffer_new(unit: libc::size_t) -> *mut hoedown_buffer;
-    fn hoedown_buffer_put(b: *mut hoedown_buffer, c: *const libc::c_char,
-                          n: libc::size_t);
+    /*fn hoedown_buffer_put(b: *mut hoedown_buffer, c: *const libc::c_char,


Why leave this in but commented out?

steveklabnik · Mar 8, 2017

src/librustdoc/html/markdown.rs

@@ -594,7 +598,7 @@ impl<'a> fmt::Display for MarkdownHtml<'a> {
 }

 pub fn plain_summary_line(md: &str) -> String {
-    extern fn link(_ob: *mut hoedown_buffer,
+    /*extern fn link(_ob: *mut hoedown_buffer,


frewsxcv · Mar 8, 2017

frewsxcv reviewed Mar 8, 2017

View changes

frewsxcv · Mar 8, 2017

src/librustdoc/html/markdown.rs

+    let mut register_header = None;
+    'main: loop {
+        let next_event = parser.next();
+        if let Some(event) = next_event {


If you made this:

let event = match next_event { Some(event) => event, None => break, }; ....

This would reduce the indentation

At that point, couldn't this be a:

while let Some(event) = parser.next() { ... }

steveklabnik · Mar 8, 2017

steveklabnik reviewed Mar 8, 2017

View changes

Seems good to me, though I agree with @frewsxcv 's comments about nesting

GuillaumeGomez · Mar 10, 2017

This is now ready to review!

frewsxcv · Mar 10, 2017

frewsxcv reviewed Mar 10, 2017

View changes

frewsxcv · Mar 10, 2017

src/librustdoc/html/markdown.rs

+        let mut content = String::new();
+        loop {
+            let event = parser.next();
+            if let Some(event) = event {


Could this could just be a for loop?

Agreed, this is for event in parser { match event { … } } (see main review comment)

frewsxcv · Mar 10, 2017

frewsxcv reviewed Mar 10, 2017

View changes

frewsxcv · Mar 10, 2017

src/librustdoc/html/markdown.rs

+    fn link(parser: &mut Parser, buffer: &mut String, url: &str, mut title: String) {
+        loop {
+            let event = parser.next();
+            if let Some(event) = event {


for loop?

frewsxcv · Mar 10, 2017

frewsxcv reviewed Mar 10, 2017

View changes

frewsxcv · Mar 10, 2017

src/librustdoc/html/markdown.rs

+        let mut content = String::new();
+        loop {
+            let event = parser.next();
+            if let Some(event) = event {


for loop?

frewsxcv · Mar 10, 2017

frewsxcv reviewed Mar 10, 2017

View changes

frewsxcv · Mar 10, 2017

src/librustdoc/html/markdown.rs

 use std::fmt::{self, Write};
-use std::slice;
+//use std::slice;


Remove commented out imports

frewsxcv · Mar 10, 2017

src/librustdoc/html/markdown.rs

 /// A unit struct which has the `fmt::Display` trait implemented. When
 /// formatted, this struct will emit the HTML corresponding to the rendered
 /// version of the contained markdown string.
-pub struct Markdown<'a>(pub &'a str);
+// The second parameter is whether we need a shorter version or not.


What is a "shorter version"?

Maybe this should be a struct with named fields instead of using unnamed fields

yeah, boolean parameters are usually a code smell

I think I'd prefer an enum MarkdownOutputStyle { Compact, Fancy } (or some other fitting variant names) instead of bool

@killercup: Good idea!

frewsxcv · Mar 10, 2017

src/librustdoc/html/markdown.rs

@@ -26,13 +26,13 @@

 #![allow(non_camel_case_types)]

-use libc;
+//use libc;


Do we still use the libc crate in rustdoc?

frewsxcv · Mar 10, 2017

src/librustdoc/html/markdown.rs

+                    (b'0' <= c && c <= b'9') ||
+                    c == b'-' || c == b'_' || c == b'.' ||
+                    c == b'~' || c == b'!' || c == b'\'' ||
+                    c == b'(' || c == b')' || c == b'*'


Pull in rust-url for this?

This is not part of the change, however I can do it next up.

frewsxcv · Mar 10, 2017

src/librustdoc/html/markdown.rs

    }

+    fn looper<'a>(parser: &'a mut Parser, buffer: &mut String, next_event: Option<Event<'a>>,
+                  toc_builder: &mut Option<TocBuilder>, shorter: bool) -> bool {
+        if let Some(event) = next_event {


Could return early here if this is None

Doesn't change much from my point of view... I think this syntax is actually better.

steveklabnik · Mar 10, 2017

/cc @raphlinus ❤️

steveklabnik · Mar 10, 2017

So, I'm spot-checking some text to find regressions.

std::collections module page

std::convert module page

prelude:

some smooshed text

std::sync::LockResult

std::string

that's what i found just now. we may need to tweak some text, and it's probably good to go through the whole entire thing once the technical side is good and clear up these kinds of issues.

killercup · Mar 10, 2017

killercup reviewed Mar 10, 2017 • edited

View changes

Nice to see some progress here!

One pattern I immediately saw was all the loop constructs matching the event and tag. I'd:

use Event::* and use Tag::*; to reduce the visual noise
Try to rewrite them as let events = parser.$filter_relevant_events; for event in events { match event {…} }, where $filter_relevant_events are some operations on the iterator to get the events you are interested in (e.g., to use take_while instead of adding break arms).

FYI, I wrote a macro once for this exact use case to allow events.take_while(not!(end Tag::Paragraph)) and similar. This may help converting most of these loops into nice iterators.

You can find it and example using it here. You'll probably need to extend it to support multiple end tag variants as used below.

I'd also really try to get rid of the shorter: bool stuff and replace with some semantically meaningful names :)

Switching parser is not something I expect to just work, even when both apparently conform to the CommonMark spec. Before merging this, I'd render a bunch of docs (std and some larger crates), and try to

set up a script that opens and takes screenshots of all generated html pages to visually diff them with ones generated by the current stable rustdoc,
or let humans look at the rendered pages,
or do both 😄

killercup · Mar 10, 2017

src/Cargo.lock

@@ -1203,6 +1203,7 @@ dependencies = [
 "build_helper 0.1.0",
 "gcc 0.3.43 (registry+https://github.com/rust-lang/crates.io-index)",
 "log 0.0.0",
+ "pulldown-cmark 0.0.8 (registry+https://github.com/rust-lang/crates.io-index)",


0.0.8… Another lib we should help get to 1.0

killercup · Mar 10, 2017

src/librustdoc/html/markdown.rs

            }
        }
+        buffer.push_str(&format!("<table>{}{}</table>",
+                                 content,
+                                 if shorter || rows.is_empty() {


Is this the only place where shorter is actually used? The name shorter does not imply "no tbody tag" to me…

In paragraph as well. The point is to display only the first line, so everything that could be on more than one line use it.

killercup · Mar 10, 2017

src/librustdoc/html/markdown.rs

 /// A unit struct which has the `fmt::Display` trait implemented. When
 /// formatted, this struct will emit the HTML corresponding to the rendered
 /// version of the contained markdown string.
-pub struct Markdown<'a>(pub &'a str);
+// The second parameter is whether we need a shorter version or not.


I think I'd prefer an enum MarkdownOutputStyle { Compact, Fancy } (or some other fitting variant names) instead of bool

killercup · Mar 10, 2017

src/librustdoc/html/markdown.rs

+        let mut content = String::new();
+        loop {
+            let event = parser.next();
+            if let Some(event) = event {


Agreed, this is for event in parser { match event { … } } (see main review comment)

killercup · Mar 10, 2017

src/librustdoc/html/markdown.rs

+                            _ => {}
+                        }
+                    } else {
+                        break 'main;


finally, a good reason not use a for loop ;)

GuillaumeGomez · Mar 11, 2017

Most problems should be solved now. I'll write a macro next to allow to remove all these loops.

ollie27 · Mar 11, 2017

Is there any reason you're not using pulldown_cmark::html::push_html to actually render the HTML?

bors · Mar 11, 2017

☔️ The latest upstream changes (presumably #40432) made this pull request unmergeable. Please resolve the merge conflicts.

GuillaumeGomez · Mar 11, 2017

@ollie27: Unfortunately yes: the generation of urls mainly (for play.rust-lang in blockcodes).

steveklabnik · Mar 11, 2017

steveklabnik reviewed Mar 11, 2017

View changes

steveklabnik · Mar 11, 2017

src/librustdoc/html/markdown.rs

-                            "&lt;", "&gt;", "&amp;", "&#39;", "&quot;"];
-        for sub in repl_sub {
-            id = id.replace(sub, "");
-        }


shouldn't this mean the comment above goes away too?

Absolutely!

killercup · Mar 11, 2017

killercup reviewed Mar 11, 2017

View changes

Code looks nicer with the macro, though I'd probably make it more generic.

Also, why not use Event::* and use Tag::*?

killercup · Mar 11, 2017

src/librustdoc/html/markdown.rs

@@ -104,6 +104,25 @@ thread_local!(pub static PLAYGROUND: RefCell<Option<(Option<String>, String)>> =
    RefCell::new(None)
 });

+macro_rules! event_loop_break {


Nice. I'd call it collect_md_events, though, and make calling it a bit more expressive, e.g.

collect_md_events!(from parser into &mut buf (toc_builder, shorter) until End(Header(_)))

Not sure if this is super useful. Anyone else has an opinion on it?

killercup · Mar 11, 2017

src/librustdoc/html/markdown.rs

+              shorter: MarkdownOutputStyle) -> fmt::Result {
+    fn block(parser: &mut Parser, buffer: &mut String, lang: &str) {
+        let mut origtext = String::new();
+        while let Some(event) = parser.next() {


Could also be using the macro (e.g. when calling without toc builder and shorter)

I didn't see much interest in expanding the macro to handle this case. But yes it's possible.

GuillaumeGomez · Mar 12, 2017

Also, why not use Event::* and use Tag::*?

Simply because I like the code being explicit. At least with this syntax, you don't have to think about it, the information is already there.

ollie27 · Mar 12, 2017

@ollie27: Unfortunately yes: the generation of urls mainly (for play.rust-lang in blockcodes).

You can still use push_html and add these links by implementing an iterator adapter around Parser. It would pass most Events through but you could filter and modify the ones you're interested in like CodeBlock and Header. This way you don't have to reimplement all the HTML rendering that pulldown-cmark already provides.

GuillaumeGomez · Mar 28, 2017

Updated.

frewsxcv · Mar 28, 2017

frewsxcv approved these changes Mar 28, 2017

View changes

I have a bunch of nitpicky style things, but they aren't important enough to hold up this PR any longer. Looks good to me!

GuillaumeGomez · Mar 29, 2017

@frewsxcv, @steveklabnik: Since both of you agreed on the changes, I'll r+ this PR. Don't hesitate to r- if you find anything!

@bors: r=frewsxcv,steveklabnik

bors · Mar 29, 2017

📌 Commit a7c6d3e has been approved by frewsxcv,steveklabnik

bors · Mar 29, 2017

⌛️ Testing commit a7c6d3e with merge abf5592...

bors · Mar 29, 2017

☀️ Test successful - status-appveyor, status-travis
Approved by: frewsxcv,steveklabnik
Pushing abf5592 to master...

ollie27 · Mar 29, 2017

Was this actually finished? It's missing support for horizontal rules, footnotes and images. There are also several bugs like missing the <br /> for HardBreaks, the title part of links being rendered as visible text rather than as an attribute on the <a> tag and MarkdownHtml now doing the same as Markdown when it should be escaping raw HTML.

As I previously mentioned, using push_html would have been much easier than trying to reimplement all of the HTML rendering.

frewsxcv · Mar 29, 2017

Was this actually finished? It's missing support for horizontal rules, footnotes and images. There are also several bugs like missing the <br /> for HardBreaks, the title part of links being rendered as visible text rather than as an attribute on the <a> tag and MarkdownHtml now doing the same as Markdown when it should be escaping raw HTML.

Have you confirmed any of these? I can bring up these concerns during today's Documentation Team meeting.

ollie27 · Mar 29, 2017

Have you confirmed any of these?

Yes. Try rendering the following in the current rustdoc and after this change:

/// markdown test
///
/// this is a [link].
///
/// [link]: https://example.com "this is a title"
///
/// hard break:  
/// after hard break
///
/// -----------
///
/// a footnote[^footnote].
///
/// [^footnote]: Thing
///
/// ![Rust](https://www.rust-lang.org/logos/rust-logo-128x128-blk-v2.png)
#[deprecated(note = "Struct<T>")]
pub fn f() {}

Before:

After:

steveklabnik · Mar 29, 2017

Two things:

this landed after last night's nightly, so tonight's is the first night where it lands. I was already planning on trying to publicize "please look at this since it's a big change", so knowing this stuff is good.
I am currently building docs with the commit before and the commit after, and then i'm going to diff them to see exactly what's different.

I don't think that this missing a few things is the end of the world; now that this has landed, fixing stuff up is much easier. We've still got plenty of time before the release; four weeks until beta, and then we could backport anything catastrophic. Getting this out in the wild sooner rather than later is important IMHO.

steveklabnik · Mar 29, 2017

Here is an un-semantic diff steveklabnik/docdiff@5d17061

steveklabnik · Mar 29, 2017

A much prettier diff steveklabnik/docdiff@HEAD~1...master

steveklabnik · Mar 29, 2017

I've filed #40912 to track these regressions. 😄

mbrubeck · Sep 28, 2017

This was disabled by default in #41431, and #44229 is tracking the plan to enable it by default.

rust-highfive assigned steveklabnik Mar 8, 2017

GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch from 9268ec6 to 2c33d62 Mar 8, 2017

GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch 4 times, most recently from 376b5dd to 2537b41 Mar 8, 2017

GuillaumeGomez changed the title from [WIP] Replace hoedown with pull in rustdoc to Replace hoedown with pull in rustdoc Mar 10, 2017

GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch 2 times, most recently from c132361 to 8b61716 Mar 11, 2017

GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch from 3c6a31f to 7f190fe Mar 12, 2017

Improve function naming

Loading status checks…

a7c6d3e

GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch from 83972b3 to a7c6d3e Mar 28, 2017

bors added a commit that referenced this pull request Mar 29, 2017

Auto merge of #40338 - GuillaumeGomez:pulldown-switch, r=frewsxcv,ste…
…veklabnik Replace hoedown with pull in rustdoc cc @rust-lang/docs

Loading status checks…

abf5592

bors merged commit a7c6d3e into rust-lang:master Mar 29, 2017
2 checks passed

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

homu Test successful
Details

GuillaumeGomez deleted the GuillaumeGomez:pulldown-switch branch Mar 29, 2017

steveklabnik referenced this pull request Mar 29, 2017
Closed
Tracking issue for hoedown -> pulldown regressions #40912
13 of 13 tasks complete

frewsxcv referenced this pull request Mar 29, 2017
Closed
Migrate from hoedown to pulldown-cmark for markdown parsing #38400

SergioBenitez referenced this pull request Mar 30, 2017
Closed
Panic caused by Markdown in latest nightly #40918

jdm referenced this pull request Mar 31, 2017
Closed
Docs build is failing #16213

steveklabnik added the relnotes label Apr 1, 2017

nrc referenced this pull request Sep 28, 2017
Closed
Tracking issue (Rustdoc): hoedown -> pulldown migration #44229
10 of 10 tasks complete

QuietMisdreavus referenced this pull request Feb 16, 2018
Merged
Remove hoedown from rustdoc #48274

rust-lang/rust

Join GitHub today

Replace hoedown with pull in rustdoc #40338

Conversation

GuillaumeGomez commented Mar 8, 2017 • edited

rust-highfive assigned steveklabnik Mar 8, 2017

This comment has been minimized.

rust-highfive commented Mar 8, 2017

steveklabnik reviewed Mar 8, 2017 View changes

This comment has been minimized.

This comment has been minimized.

frewsxcv reviewed Mar 8, 2017 View changes

This comment has been minimized.

This comment has been minimized.

GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch from 9268ec6 to 2c33d62 Mar 8, 2017

steveklabnik reviewed Mar 8, 2017 View changes

GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch 4 times, most recently from 376b5dd to 2537b41 Mar 8, 2017

GuillaumeGomez changed the title from [WIP] Replace hoedown with pull in rustdoc to Replace hoedown with pull in rustdoc Mar 10, 2017

This comment has been minimized.

GuillaumeGomez commented Mar 10, 2017

frewsxcv reviewed Mar 10, 2017 View changes

This comment has been minimized.

This comment has been minimized.

killercup Mar 10, 2017 • edited

frewsxcv reviewed Mar 10, 2017 View changes

This comment has been minimized.

frewsxcv reviewed Mar 10, 2017 View changes

This comment has been minimized.

frewsxcv reviewed Mar 10, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

steveklabnik commented Mar 10, 2017

This comment has been minimized.

steveklabnik commented Mar 10, 2017

killercup reviewed Mar 10, 2017 • edited View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

killercup Mar 10, 2017 • edited

This comment has been minimized.

This comment has been minimized.

GuillaumeGomez commented Mar 11, 2017

This comment has been minimized.

ollie27 commented Mar 11, 2017

This comment has been minimized.

bors commented Mar 11, 2017

This comment has been minimized.

GuillaumeGomez commented Mar 11, 2017 • edited

steveklabnik reviewed Mar 11, 2017 View changes

This comment has been minimized.

This comment has been minimized.

GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch 2 times, most recently from c132361 to 8b61716 Mar 11, 2017

killercup reviewed Mar 11, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch from 3c6a31f to 7f190fe Mar 12, 2017

This comment has been minimized.

GuillaumeGomez commented Mar 12, 2017

This comment has been minimized.

ollie27 commented Mar 12, 2017 • edited

GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch from 83972b3 to a7c6d3e Mar 28, 2017

This comment has been minimized.

GuillaumeGomez commented Mar 28, 2017

frewsxcv approved these changes Mar 28, 2017 View changes

This comment has been minimized.

GuillaumeGomez commented Mar 29, 2017

This comment has been minimized.

GuillaumeGomez commented Mar 8, 2017 •

edited

steveklabnik reviewed Mar 8, 2017

View changes

frewsxcv reviewed Mar 8, 2017

View changes

GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch from `9268ec6` to `2c33d62` Mar 8, 2017

steveklabnik reviewed Mar 8, 2017

View changes

GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch 4 times, most recently from `376b5dd` to `2537b41` Mar 8, 2017

frewsxcv reviewed Mar 10, 2017

View changes

killercup Mar 10, 2017 •

edited

frewsxcv reviewed Mar 10, 2017

View changes

frewsxcv reviewed Mar 10, 2017

View changes

frewsxcv reviewed Mar 10, 2017

View changes

killercup reviewed Mar 10, 2017 • edited

View changes

killercup Mar 10, 2017 •

edited

GuillaumeGomez commented Mar 11, 2017 •

edited

steveklabnik reviewed Mar 11, 2017

View changes

GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch 2 times, most recently from `c132361` to `8b61716` Mar 11, 2017

killercup reviewed Mar 11, 2017

View changes

GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch from `3c6a31f` to `7f190fe` Mar 12, 2017

ollie27 commented Mar 12, 2017 •

edited

GuillaumeGomez force-pushed the GuillaumeGomez:pulldown-switch branch from `83972b3` to `a7c6d3e` Mar 28, 2017

frewsxcv approved these changes Mar 28, 2017

View changes

bors merged commit `a7c6d3e` into rust-lang:master Mar 29, 2017
2 checks passed