Hoedown big comeback!

[GuillaumeGomez]

> cargo +local test
   Compiling libc v0.2.20
   Compiling sysinfo v0.3.4 (file:///Users/imperio/rust/sysinfo)
    Finished dev [unoptimized + debuginfo] target(s) in 3.2 secs
     Running target/debug/deps/disk_list-dbd70897f1f7e080

running 1 test
test test_disks ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured

     Running target/debug/deps/sysinfo-8ad11103abdf5941

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured

   Doc-tests sysinfo
WARNING: src/sysinfo.rs -  (line 45) test will be run in the next rustdoc version. If it's not supposed to, please update your documentation and make it compliant to common mark specifications.
WARNING: src/sysinfo.rs -  (line 48) test will be run in the next rustdoc version. If it's not supposed to, please update your documentation and make it compliant to common mark specifications.

running 1 test
test src/sysinfo.rs -  (line 14) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured

r? @rust-lang/docs

[QuietMisdreavus]

🎊 🎉 🎊

Should this get a crater/cargobomb run, to make sure we caught all the "bad tests" that are getting run in the pulldown era?

Also, Travis choked when it tried to run tidy on the hoedown source.

[QuietMisdreavus]

🎊 🎉 🎊

Should this get a crater/cargobomb run, to make sure we caught all the "bad tests" that are getting run in the pulldown era?

Also, Travis choked when it tried to run tidy on the hoedown source.

[QuietMisdreavus]

I may not have been present for the discussion, but this leaves a question: What's the timeline for this changeover now? Are we expecting this to follow some kind of stabilization FCP process to turn the commonmark tests back on?

There's also the matter of how loud this might need to be. I wonder how many people manually run the test suites on their projects, or just leave it to CI. If CI reports success even though these warnings come out, how much progress have we made?

[GuillaumeGomez]

@steveklabnik said "a few releases". And the point is to prevent to avoid current massive breakage. After that, we remove the warning and the tests will fail.

[QuietMisdreavus]

If that's the case, and we're leaving this on long enough, then I think it'll be fine, as long as we raise a big enough clamor about it one way or another. We may want the warning text to say something like "a later rustdoc version", though, since it's not technically the "next" one.

[QuietMisdreavus]

I take it this is very similar to the previous hoedown "find testable code" function, just updated to spit them into old_tests instead?

[GuillaumeGomez]

Absolutely. A bit shorter.

[GuillaumeGomez]

Tidy fixed.

[steveklabnik]

Should this get a crater/cargobomb run, to make sure we caught all the "bad tests" that are getting run in the pulldown era?

This would be great, but @brson can't even complete the regular cargobomb run until next week. At that point, we already have a release going. So, I'm not sure :( (I couldn't get cargobomb working personally)

[steveklabnik]

There was a PR for COPYRIGHT that needs to be brought back too

[steveklabnik]

review note: this is the same branch as was here previously. leaving a comment for myself so I don't forget that I checked!

[steveklabnik]

This overall looks good to me. Two things:

1. We should provide a flag of some kind to run the test suite under pulldown-cmark's parsing, so that people can ensure things work.

2. I'd like to bikeshed the message though. Here it is today:

WARNING: src/sysinfo.rs -  (line 48) test will be run in the next rustdoc version. If it's not supposed to, please update your documentation and make it compliant to common mark specifications.

I think I would write

WARNING: src/sysinfo.rs -  (line 48) Code block is not currently run as a test, but will in future versions of rustdoc. Pass the `--use-commonmark` flag to try out this behavior, and make sure this test will pass.

Or something along those lines. I really wish we had a way to provide more and better information for people to help ease the transition, idk.

@rust-lang/compiler you have some experience with this kind of thing, any suggestions?

[mrhota]

I don't understand; what's the point of this PR? didn't we replace hoedown with pulldown-cmark?

[steveklabnik]

@mrhota

The issue is, the change caused many things that weren't tested as rust code before to be tested now, and those tests failed.

Take this:

# A heading
```
  not rust code
```

with hoedown, this was not considered a test, because of the way it was parsed. With pulldown-cmark, it is properly identified as a codeblock, and therefore, Rust code, and so should run as a test. But the example was invalid, and so, the tests failed.

When we announced the change, nobody really commented, or rather, we had four or five crates that needed updating to fix this. But a cargobomb run identified a lot of crates, way more than we are willing to regress on, that ended up failing their test suites because of this kind of thing. So we need to spread this out over a longer period of time than just one release.

Does that make sense?

[mrhota]

yes, completely! I didn't even know this was a problem. yikes

[steveklabnik]

Update: I don't think the flag is needed, as once your stuff runs without any warnings, you're good. So maybe focus the error message on that?

WARNING: src/sysinfo.rs - (line 48) Code block is not currently run as a test, but will in future versions of rustdoc. Please ensure this code block is a runnable test, or use the ignore directive.

[steveklabnik]

@bors: r+

thanks!

[bors]

📌 Commit bee0291 has been approved by steveklabnik

[alexcrichton]

@GuillaumeGomez @steveklabnik

My personal hope was that we would bring hoedown back in full as part of this, not just for finding testable code. I realize that the only visible regressions are those related to doctests but are there not a large number of rendering regressions as well? Both related to changing to CommonMark but also bugs in the implementation?

My assumption would be that the new pulldown-cmark implementation would be behind a flag by default. We would print warnings indicating breakage on doctests immediately and otherwise strongly encourage authors to test their documentation rendering (via something like RUSTDOCFLAGS to Cargo). Then eventually we can switch the defaults (still issuing warnings) and then finally delete hoedown at a later date as warnings turn into hard errors.

I fear that this PR currently only fixes cargobomb, and still leaves the door open to a lot of breakage wrt rendering.

[GuillaumeGomez]

@alexcrichton: If we start putting this change behind a flag, no one will use it and the time of adoption will only grow. I was against putting back hoedown (even for this case) and more in favor for a hard break.

[nikomatsakis]

@GuillaumeGomez there is some truth to that -- I agree that most people will not opt in, even if we advertise the change widely. Nonetheless, the "best case" procedure is to go through several stages:

• keep old behavior, but warn that new behavior is coming;
• bring in new behavior by default, but leave an opt out;
• remove old behavior.

Very often, this requires keeping two copies of the code around (one that does the old thing, one that does the new thing), and then diffing the results. In some cases, it's just not possible to get precise warnings -- in that case, we typically try very hard to do warnings where we can. In very few cases, we skip the early stages because there doesn't seem to be much effect in practice.

It is true that people often ignore the warnings. But doing it in several stages has the advantage that we at least give people a chance to respond, which means that when the breakage finally comes, it is less frustrating (you know it was coming, after all, or at least you can see that we made an effort to let you know). In short, breaking changes always disturb people, and the best thing is to disturb them as little as we can (btw, if we can come up with ways to do this better and/or with less effort, I'm all ears! Avoiding breakage is a constant struggle.)

The other advantage of warnings is that we can sometimes detect them automatically, of course, and hence we can offer up PRs or at least ping people to bring their attention to the fact that warnings will be transitioning into harder errors.

[nikomatsakis]

(An example of this procedure is the forwards compatibility lints, which start out as warnings, proceed to deny-by-default, and then make a hard error.)