Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

regression: doctest failure display-as-0.4.4 #61562

Closed
Mark-Simulacrum opened this issue Jun 5, 2019 · 8 comments
Closed

regression: doctest failure display-as-0.4.4 #61562

Mark-Simulacrum opened this issue Jun 5, 2019 · 8 comments
Labels
A-doctests Area: Documentation tests, run by rustdoc regression-from-stable-to-stable Performance or correctness regression from one stable version to another. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue.

Comments

@Mark-Simulacrum
Copy link
Member

Behavior change from stable to beta but I have not investigated or tried to narrow down to anything more specific. Possibly a rustdoc problem (failure is in a doc test) so tagging with T-rustdoc.
@Mark-Simulacrum Mark-Simulacrum added T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. regression-from-stable-to-beta Performance or correctness regression from stable to beta. labels Jun 5, 2019
@QuietMisdreavus
Copy link
Member

This is strange; it looks like the behavior for detecting doctests changed? What's the commit range for stable...beta? I don't remember any major changes to doctest detection, but i haven't looked into that yet.

[misdreavus@tonberry display-as]$ cargo +stable test --doc                                                                                    [17/1140]
    Finished dev [unoptimized + debuginfo] target(s) in 0.28s
   Doc-tests display-as

running 8 tests
test src/lib.rs -  (line 113) ... ok
test src/lib.rs -  (line 149) ... ignored
test src/lib.rs -  (line 100) ... ok
test src/lib.rs -  (line 127) ... ok
test src/html.rs - display_floats_as (line 78) ... ok
test src/lib.rs -  (line 58) ... ok
test src/lib.rs -  (line 72) ... ok
test src/lib.rs -  (line 85) ... ok

test result: ok. 7 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out

[misdreavus@tonberry display-as]$ cargo +local test --doc
    Finished dev [unoptimized + debuginfo] target(s) in 0.18s
   Doc-tests display-as

running 12 tests                                                                                                                                       test src/lib.rs -  (line 104) ... ok
test src/lib.rs -  (line 156) ... ignored                                                                                                              test src/lib.rs -  (line 178) ... ignored
test src/lib.rs -  (line 183) ... ignored
test src/lib.rs -  (line 188) ... ignored
test src/lib.rs -  (line 133) ... ok
test src/lib.rs -  (line 118) ... ok
test src/html.rs - display_floats_as (line 78) ... ok
test src/lib.rs -  (line 74) ... ok
test src/lib.rs -  (line 59) ... ok
test src/lib.rs -  (line 88) ... ok
test src/lib.rs -  (line 193) ... FAILED

failures:

---- src/lib.rs -  (line 193) stdout ----
thread 'src/lib.rs -  (line 193)' panicked at 'test executable failed:

thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `"<title>PH365: Templates</title>\n<html>\n  <ul><li><span class=\"student\">Name: David</span>\n\n</li><li><span class=\"student\">Name: Joel<
/span>\n\n</li></ul>\n</html>\n\n\n"`,
 right: `"<title>PH365: Templates</title>\n<html>\n  <ul>\n\n  // This is buggy:  I want to iterate, but it fails!\n  for s in self.students.iter() {\n
    \"<li>\" s \"</li>\"\n  }\n\n  </ul>\n</html>\n\n\n"`', src/lib.rs:18:1
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

', src/librustdoc/test.rs:342:17
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.


failures:
    src/lib.rs -  (line 193)

test result: FAILED. 7 passed; 1 failed; 4 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--doc'

The doctest in question (the one at the very end of the crate docs) seems like it's not being run at all on stable. When i extracted the doctest into a standalone crate, it failed in the same way when i built it with stable.

@QuietMisdreavus
Copy link
Member

It looks like this was caused by #60140 giving pulldown the ability to detect more doctests. It looks like this doctest was accidentally ignored by the older version of pulldown, which the crate author didn't notice before publishing. (The repo also hasn't been touched after the new pulldown was enabled, which is probably why this wasn't noticed.)

@rust-lang/rustdoc Since this doesn't seem to have affected many crates, it's possible we can let this slide, but the alternative is we roll back the pulldown upgrade and have another warning period. Just like when we introduced pulldown in the first place.

@GuillaumeGomez
Copy link
Member

It's a fix so I wonder if this applies the same way...

@droundy
Copy link
Contributor

droundy commented Jun 10, 2019

Any idea why it isn't being run in the stable compiler, and how I could fix that? The comment indicates that I expected a bug (it's vague in my memory) but I'd like to fix the test.

@QuietMisdreavus
Copy link
Member

@droundy It looks like the code blocks where you linked in the template files are empty when --features docinclude is missing, which might have confused rustdoc. I changed the ignore to text and stable rustdoc was able to skip past those and run the doctest at the end of the crate docs.

@pnkfelix
Copy link
Member

@Mark-Simulacrum does this need to be tagged T-compiler? Seems like a pure rustdoc issue to me, but maybe I misunderstand?

@Mark-Simulacrum Mark-Simulacrum added regression-from-stable-to-stable Performance or correctness regression from one stable version to another. and removed T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. regression-from-stable-to-beta Performance or correctness regression from stable to beta. labels Jul 11, 2019
@lqd lqd mentioned this issue Oct 3, 2019
Closed
@jyn514 jyn514 added the A-doctests Area: Documentation tests, run by rustdoc label Aug 25, 2020
@jyn514
Copy link
Member

jyn514 commented Aug 25, 2020

It looks like this doctest was accidentally ignored by the older version of pulldown, which the crate author didn't notice before publishing.

So this isn't a bug in rustdoc at all, right? This is a bug that we fixed, and the failure was pre-existing?

@jyn514
Copy link
Member

jyn514 commented Nov 12, 2020

Any idea why it isn't being run in the stable compiler, and how I could fix that? The comment indicates that I expected a bug (it's vague in my memory) but I'd like to fix the test.

Since the change is fixing a bug and the backwards incompatible change landed almost 18 months ago, I'm going to close this.

@jyn514 jyn514 closed this as completed Nov 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-doctests Area: Documentation tests, run by rustdoc regression-from-stable-to-stable Performance or correctness regression from one stable version to another. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@droundy @pnkfelix @GuillaumeGomez @Mark-Simulacrum @QuietMisdreavus @jyn514