Replace hoedown with pull in rustdoc #40338
Conversation
(rust_highfive has picked a reviewer for you, use r? to override) |
I will give this a real review tomorrow, but |
@@ -179,8 +181,8 @@ extern { | |||
fn hoedown_document_free(md: *mut hoedown_document); | |||
|
|||
fn hoedown_buffer_new(unit: libc::size_t) -> *mut hoedown_buffer; | |||
fn hoedown_buffer_put(b: *mut hoedown_buffer, c: *const libc::c_char, | |||
n: libc::size_t); | |||
/*fn hoedown_buffer_put(b: *mut hoedown_buffer, c: *const libc::c_char, |
steveklabnik
Mar 8, 2017
Member
Why leave this in but commented out?
Why leave this in but commented out?
@@ -594,7 +598,7 @@ impl<'a> fmt::Display for MarkdownHtml<'a> { | |||
} | |||
|
|||
pub fn plain_summary_line(md: &str) -> String { | |||
extern fn link(_ob: *mut hoedown_buffer, | |||
/*extern fn link(_ob: *mut hoedown_buffer, |
steveklabnik
Mar 8, 2017
Member
same here
same here
let mut register_header = None; | ||
'main: loop { | ||
let next_event = parser.next(); | ||
if let Some(event) = next_event { |
frewsxcv
Mar 8, 2017
Member
If you made this:
let event = match next_event {
Some(event) => event,
None => break,
};
....
This would reduce the indentation
If you made this:
let event = match next_event {
Some(event) => event,
None => break,
};
....
This would reduce the indentation
frewsxcv
Mar 8, 2017
Member
At that point, couldn't this be a:
while let Some(event) = parser.next() {
...
}
At that point, couldn't this be a:
while let Some(event) = parser.next() {
...
}
9268ec6
to
2c33d62
Seems good to me, though I agree with @frewsxcv 's comments about nesting |
376b5dd
to
2537b41
This is now ready to review! |
let mut content = String::new(); | ||
loop { | ||
let event = parser.next(); | ||
if let Some(event) = event { |
frewsxcv
Mar 10, 2017
Member
Could this could just be a for
loop?
Could this could just be a for
loop?
killercup
Mar 10, 2017
•
Member
Agreed, this is for event in parser { match event { … } }
(see main review comment)
Agreed, this is for event in parser { match event { … } }
(see main review comment)
fn link(parser: &mut Parser, buffer: &mut String, url: &str, mut title: String) { | ||
loop { | ||
let event = parser.next(); | ||
if let Some(event) = event { |
frewsxcv
Mar 10, 2017
Member
for
loop?
for
loop?
let mut content = String::new(); | ||
loop { | ||
let event = parser.next(); | ||
if let Some(event) = event { |
frewsxcv
Mar 10, 2017
Member
for
loop?
for
loop?
use std::fmt::{self, Write}; | ||
use std::slice; | ||
//use std::slice; |
frewsxcv
Mar 10, 2017
Member
Remove commented out imports
Remove commented out imports
/// A unit struct which has the `fmt::Display` trait implemented. When | ||
/// formatted, this struct will emit the HTML corresponding to the rendered | ||
/// version of the contained markdown string. | ||
pub struct Markdown<'a>(pub &'a str); | ||
// The second parameter is whether we need a shorter version or not. |
frewsxcv
Mar 10, 2017
Member
What is a "shorter version"?
Maybe this should be a struct with named fields instead of using unnamed fields
What is a "shorter version"?
Maybe this should be a struct with named fields instead of using unnamed fields
steveklabnik
Mar 10, 2017
Member
yeah, boolean parameters are usually a code smell
yeah, boolean parameters are usually a code smell
killercup
Mar 10, 2017
Member
I think I'd prefer an enum MarkdownOutputStyle { Compact, Fancy }
(or some other fitting variant names) instead of bool
I think I'd prefer an enum MarkdownOutputStyle { Compact, Fancy }
(or some other fitting variant names) instead of bool
@@ -26,13 +26,13 @@ | |||
|
|||
#![allow(non_camel_case_types)] | |||
|
|||
use libc; | |||
//use libc; |
frewsxcv
Mar 10, 2017
Member
Do we still use the libc crate in rustdoc?
Do we still use the libc crate in rustdoc?
(b'0' <= c && c <= b'9') || | ||
c == b'-' || c == b'_' || c == b'.' || | ||
c == b'~' || c == b'!' || c == b'\'' || | ||
c == b'(' || c == b')' || c == b'*' |
frewsxcv
Mar 10, 2017
Member
Pull in rust-url for this?
Pull in rust-url for this?
GuillaumeGomez
Mar 10, 2017
Author
Member
This is not part of the change, however I can do it next up.
This is not part of the change, however I can do it next up.
} | ||
|
||
fn looper<'a>(parser: &'a mut Parser, buffer: &mut String, next_event: Option<Event<'a>>, | ||
toc_builder: &mut Option<TocBuilder>, shorter: bool) -> bool { | ||
if let Some(event) = next_event { |
frewsxcv
Mar 10, 2017
Member
Could return early here if this is None
Could return early here if this is None
GuillaumeGomez
Mar 10, 2017
Author
Member
Doesn't change much from my point of view... I think this syntax is actually better.
Doesn't change much from my point of view... I think this syntax is actually better.
/cc @raphlinus |
So, I'm spot-checking some text to find regressions. std::collections module page std::convert module page prelude: some smooshed text std::sync::LockResult std::string that's what i found just now. we may need to tweak some text, and it's probably good to go through the whole entire thing once the technical side is good and clear up these kinds of issues. |
Nice to see some progress here! One pattern I immediately saw was all the
I'd also really try to get rid of the Switching parser is not something I expect to just work, even when both apparently conform to the CommonMark spec. Before merging this, I'd render a bunch of docs (std and some larger crates), and try to
|
@@ -1203,6 +1203,7 @@ dependencies = [ | |||
"build_helper 0.1.0", | |||
"gcc 0.3.43 (registry+https://github.com/rust-lang/crates.io-index)", | |||
"log 0.0.0", | |||
"pulldown-cmark 0.0.8 (registry+https://github.com/rust-lang/crates.io-index)", |
killercup
Mar 10, 2017
Member
0.0.8… Another lib we should help get to 1.0
0.0.8… Another lib we should help get to 1.0
} | ||
} | ||
buffer.push_str(&format!("<table>{}{}</table>", | ||
content, | ||
if shorter || rows.is_empty() { |
killercup
Mar 10, 2017
Member
Is this the only place where shorter
is actually used? The name shorter
does not imply "no tbody tag" to me…
Is this the only place where shorter
is actually used? The name shorter
does not imply "no tbody tag" to me…
GuillaumeGomez
Mar 10, 2017
Author
Member
In paragraph as well. The point is to display only the first line, so everything that could be on more than one line use it.
In paragraph as well. The point is to display only the first line, so everything that could be on more than one line use it.
/// A unit struct which has the `fmt::Display` trait implemented. When | ||
/// formatted, this struct will emit the HTML corresponding to the rendered | ||
/// version of the contained markdown string. | ||
pub struct Markdown<'a>(pub &'a str); | ||
// The second parameter is whether we need a shorter version or not. |
killercup
Mar 10, 2017
Member
I think I'd prefer an enum MarkdownOutputStyle { Compact, Fancy }
(or some other fitting variant names) instead of bool
I think I'd prefer an enum MarkdownOutputStyle { Compact, Fancy }
(or some other fitting variant names) instead of bool
let mut content = String::new(); | ||
loop { | ||
let event = parser.next(); | ||
if let Some(event) = event { |
killercup
Mar 10, 2017
•
Member
Agreed, this is for event in parser { match event { … } }
(see main review comment)
Agreed, this is for event in parser { match event { … } }
(see main review comment)
_ => {} | ||
} | ||
} else { | ||
break 'main; |
killercup
Mar 10, 2017
Member
finally, a good reason not use a for loop ;)
finally, a good reason not use a for loop ;)
Most problems should be solved now. I'll write a macro next to allow to remove all these loops. |
Is there any reason you're not using |
|
@ollie27: Unfortunately yes: the generation of urls mainly (for play.rust-lang in blockcodes). |
"<", ">", "&", "'", """]; | ||
for sub in repl_sub { | ||
id = id.replace(sub, ""); | ||
} |
steveklabnik
Mar 11, 2017
Member
shouldn't this mean the comment above goes away too?
shouldn't this mean the comment above goes away too?
GuillaumeGomez
Mar 11, 2017
Author
Member
Absolutely!
Absolutely!
c132361
to
8b61716
Code looks nicer with the macro, though I'd probably make it more generic. Also, why not |
@@ -104,6 +104,25 @@ thread_local!(pub static PLAYGROUND: RefCell<Option<(Option<String>, String)>> = | |||
RefCell::new(None) | |||
}); | |||
|
|||
macro_rules! event_loop_break { |
killercup
Mar 11, 2017
Member
Nice. I'd call it collect_md_events
, though, and make calling it a bit more expressive, e.g.
collect_md_events!(from parser into &mut buf (toc_builder, shorter) until End(Header(_)))
Nice. I'd call it collect_md_events
, though, and make calling it a bit more expressive, e.g.
collect_md_events!(from parser into &mut buf (toc_builder, shorter) until End(Header(_)))
GuillaumeGomez
Mar 12, 2017
Author
Member
Not sure if this is super useful. Anyone else has an opinion on it?
Not sure if this is super useful. Anyone else has an opinion on it?
shorter: MarkdownOutputStyle) -> fmt::Result { | ||
fn block(parser: &mut Parser, buffer: &mut String, lang: &str) { | ||
let mut origtext = String::new(); | ||
while let Some(event) = parser.next() { |
killercup
Mar 11, 2017
Member
Could also be using the macro (e.g. when calling without toc builder and shorter)
Could also be using the macro (e.g. when calling without toc builder and shorter)
GuillaumeGomez
Mar 12, 2017
Author
Member
I didn't see much interest in expanding the macro to handle this case. But yes it's possible.
I didn't see much interest in expanding the macro to handle this case. But yes it's possible.
3c6a31f
to
7f190fe
Simply because I like the code being explicit. At least with this syntax, you don't have to think about it, the information is already there. |
You can still use |
83972b3
to
a7c6d3e
Updated. |
I have a bunch of nitpicky style things, but they aren't important enough to hold up this PR any longer. Looks good to me! |
@frewsxcv, @steveklabnik: Since both of you agreed on the changes, I'll r+ this PR. Don't hesitate to r- if you find anything! @bors: r=frewsxcv,steveklabnik |
|
…veklabnik Replace hoedown with pull in rustdoc cc @rust-lang/docs
|
Was this actually finished? It's missing support for horizontal rules, footnotes and images. There are also several bugs like missing the As I previously mentioned, using |
Have you confirmed any of these? I can bring up these concerns during today's Documentation Team meeting. |
Yes. Try rendering the following in the current rustdoc and after this change: /// markdown test
///
/// this is a [link].
///
/// [link]: https://example.com "this is a title"
///
/// hard break:
/// after hard break
///
/// -----------
///
/// a footnote[^footnote].
///
/// [^footnote]: Thing
///
/// 
#[deprecated(note = "Struct<T>")]
pub fn f() {} |
Two things:
I don't think that this missing a few things is the end of the world; now that this has landed, fixing stuff up is much easier. We've still got plenty of time before the release; four weeks until beta, and then we could backport anything catastrophic. Getting this out in the wild sooner rather than later is important IMHO. |
Here is an un-semantic diff steveklabnik/docdiff@5d17061 |
A much prettier diff steveklabnik/docdiff@HEAD~1...master |
I've filed #40912 to track these regressions. |
Is it really time? Have our months, no, *years* of suffering come to an end? Are we finally able to cast off the pall of Hoedown? The weight which has dragged us down for so long? ----- So, timeline for those who need to catch up: * Way back in December 2016, [we decided we wanted to switch out the markdown renderer](rust-lang#38400). However, this was put on hold because the build system at the time made it difficult to pull in dependencies from crates.io. * A few months later, in March 2017, [the first PR was done, to switch out the renderers entirely](rust-lang#40338). The PR itself was fraught with CI and build system issues, but eventually landed. * However, not all was well in the Rustdoc world. During the PR and shortly after, we noticed [some differences in the way the two parsers handled some things](rust-lang#40912), and some of these differences were major enough to break the docs for some crates. * A couple weeks afterward, [Hoedown was put back in](rust-lang#41290), at this point just to catch tests that Pulldown was "spuriously" running. This would at least provide some warning about spurious tests, rather than just breaking spontaneously. * However, the problems had created enough noise by this point that just a few days after that, [Hoedown was switched back to the default](rust-lang#41431) while we came up with a solution for properly warning about the differences. * That solution came a few weeks later, [as a series of warnings when the HTML emitted by the two parsers was semantically different](rust-lang#41991). But that came at a cost, as now rustdoc needed proc-macro support (the new crate needed some custom derives farther down its dependency tree), and the build system was not equipped to handle it at the time. It was worked on for three months as the issue stumped more and more people. * In that time, [bootstrap was completely reworked](rust-lang#43059) to change how it ordered compilation, and [the method by which it built rustdoc would change](rust-lang#43482), as well. This allowed it to only be built after stage1, when proc-macros would be available, allowing the "rendering differences" PR to finally land. * The warnings were not perfect, and revealed a few [spurious](rust-lang#44368) [differences](rust-lang#45421) between how we handled the renderers. * Once these were handled, [we flipped the switch to turn on the "rendering difference" warnings all the time](rust-lang#45324), in October 2017. This began the "warning cycle" for this change, and landed in stable in 1.23, on 2018-01-04. * Once those warnings hit stable, and after a couple weeks of seeing whether we would get any more reports than what we got from sitting on nightly/beta, [we switched the renderers](rust-lang#47398), making Pulldown the default but still offering the option to use Hoedown. And that brings us to the present. We haven't received more new issues from this in the meantime, and the "switch by default" is now on beta. Our reasoning is that, at this point, anyone who would have been affected by this has run into it already.
Is it really time? Have our months, no, *years* of suffering come to an end? Are we finally able to cast off the pall of Hoedown? The weight which has dragged us down for so long? ----- So, timeline for those who need to catch up: * Way back in December 2016, [we decided we wanted to switch out the markdown renderer](rust-lang#38400). However, this was put on hold because the build system at the time made it difficult to pull in dependencies from crates.io. * A few months later, in March 2017, [the first PR was done, to switch out the renderers entirely](rust-lang#40338). The PR itself was fraught with CI and build system issues, but eventually landed. * However, not all was well in the Rustdoc world. During the PR and shortly after, we noticed [some differences in the way the two parsers handled some things](rust-lang#40912), and some of these differences were major enough to break the docs for some crates. * A couple weeks afterward, [Hoedown was put back in](rust-lang#41290), at this point just to catch tests that Pulldown was "spuriously" running. This would at least provide some warning about spurious tests, rather than just breaking spontaneously. * However, the problems had created enough noise by this point that just a few days after that, [Hoedown was switched back to the default](rust-lang#41431) while we came up with a solution for properly warning about the differences. * That solution came a few weeks later, [as a series of warnings when the HTML emitted by the two parsers was semantically different](rust-lang#41991). But that came at a cost, as now rustdoc needed proc-macro support (the new crate needed some custom derives farther down its dependency tree), and the build system was not equipped to handle it at the time. It was worked on for three months as the issue stumped more and more people. * In that time, [bootstrap was completely reworked](rust-lang#43059) to change how it ordered compilation, and [the method by which it built rustdoc would change](rust-lang#43482), as well. This allowed it to only be built after stage1, when proc-macros would be available, allowing the "rendering differences" PR to finally land. * The warnings were not perfect, and revealed a few [spurious](rust-lang#44368) [differences](rust-lang#45421) between how we handled the renderers. * Once these were handled, [we flipped the switch to turn on the "rendering difference" warnings all the time](rust-lang#45324), in October 2017. This began the "warning cycle" for this change, and landed in stable in 1.23, on 2018-01-04. * Once those warnings hit stable, and after a couple weeks of seeing whether we would get any more reports than what we got from sitting on nightly/beta, [we switched the renderers](rust-lang#47398), making Pulldown the default but still offering the option to use Hoedown. And that brings us to the present. We haven't received more new issues from this in the meantime, and the "switch by default" is now on beta. Our reasoning is that, at this point, anyone who would have been affected by this has run into it already.
Is it really time? Have our months, no, *years* of suffering come to an end? Are we finally able to cast off the pall of Hoedown? The weight which has dragged us down for so long? ----- So, timeline for those who need to catch up: * Way back in December 2016, [we decided we wanted to switch out the markdown renderer](rust-lang#38400). However, this was put on hold because the build system at the time made it difficult to pull in dependencies from crates.io. * A few months later, in March 2017, [the first PR was done, to switch out the renderers entirely](rust-lang#40338). The PR itself was fraught with CI and build system issues, but eventually landed. * However, not all was well in the Rustdoc world. During the PR and shortly after, we noticed [some differences in the way the two parsers handled some things](rust-lang#40912), and some of these differences were major enough to break the docs for some crates. * A couple weeks afterward, [Hoedown was put back in](rust-lang#41290), at this point just to catch tests that Pulldown was "spuriously" running. This would at least provide some warning about spurious tests, rather than just breaking spontaneously. * However, the problems had created enough noise by this point that just a few days after that, [Hoedown was switched back to the default](rust-lang#41431) while we came up with a solution for properly warning about the differences. * That solution came a few weeks later, [as a series of warnings when the HTML emitted by the two parsers was semantically different](rust-lang#41991). But that came at a cost, as now rustdoc needed proc-macro support (the new crate needed some custom derives farther down its dependency tree), and the build system was not equipped to handle it at the time. It was worked on for three months as the issue stumped more and more people. * In that time, [bootstrap was completely reworked](rust-lang#43059) to change how it ordered compilation, and [the method by which it built rustdoc would change](rust-lang#43482), as well. This allowed it to only be built after stage1, when proc-macros would be available, allowing the "rendering differences" PR to finally land. * The warnings were not perfect, and revealed a few [spurious](rust-lang#44368) [differences](rust-lang#45421) between how we handled the renderers. * Once these were handled, [we flipped the switch to turn on the "rendering difference" warnings all the time](rust-lang#45324), in October 2017. This began the "warning cycle" for this change, and landed in stable in 1.23, on 2018-01-04. * Once those warnings hit stable, and after a couple weeks of seeing whether we would get any more reports than what we got from sitting on nightly/beta, [we switched the renderers](rust-lang#47398), making Pulldown the default but still offering the option to use Hoedown. And that brings us to the present. We haven't received more new issues from this in the meantime, and the "switch by default" is now on beta. Our reasoning is that, at this point, anyone who would have been affected by this has run into it already.
Is it really time? Have our months, no, *years* of suffering come to an end? Are we finally able to cast off the pall of Hoedown? The weight which has dragged us down for so long? ----- So, timeline for those who need to catch up: * Way back in December 2016, [we decided we wanted to switch out the markdown renderer](rust-lang#38400). However, this was put on hold because the build system at the time made it difficult to pull in dependencies from crates.io. * A few months later, in March 2017, [the first PR was done, to switch out the renderers entirely](rust-lang#40338). The PR itself was fraught with CI and build system issues, but eventually landed. * However, not all was well in the Rustdoc world. During the PR and shortly after, we noticed [some differences in the way the two parsers handled some things](rust-lang#40912), and some of these differences were major enough to break the docs for some crates. * A couple weeks afterward, [Hoedown was put back in](rust-lang#41290), at this point just to catch tests that Pulldown was "spuriously" running. This would at least provide some warning about spurious tests, rather than just breaking spontaneously. * However, the problems had created enough noise by this point that just a few days after that, [Hoedown was switched back to the default](rust-lang#41431) while we came up with a solution for properly warning about the differences. * That solution came a few weeks later, [as a series of warnings when the HTML emitted by the two parsers was semantically different](rust-lang#41991). But that came at a cost, as now rustdoc needed proc-macro support (the new crate needed some custom derives farther down its dependency tree), and the build system was not equipped to handle it at the time. It was worked on for three months as the issue stumped more and more people. * In that time, [bootstrap was completely reworked](rust-lang#43059) to change how it ordered compilation, and [the method by which it built rustdoc would change](rust-lang#43482), as well. This allowed it to only be built after stage1, when proc-macros would be available, allowing the "rendering differences" PR to finally land. * The warnings were not perfect, and revealed a few [spurious](rust-lang#44368) [differences](rust-lang#45421) between how we handled the renderers. * Once these were handled, [we flipped the switch to turn on the "rendering difference" warnings all the time](rust-lang#45324), in October 2017. This began the "warning cycle" for this change, and landed in stable in 1.23, on 2018-01-04. * Once those warnings hit stable, and after a couple weeks of seeing whether we would get any more reports than what we got from sitting on nightly/beta, [we switched the renderers](rust-lang#47398), making Pulldown the default but still offering the option to use Hoedown. And that brings us to the present. We haven't received more new issues from this in the meantime, and the "switch by default" is now on beta. Our reasoning is that, at this point, anyone who would have been affected by this has run into it already.
cc @rust-lang/docs