coverage: `llvm-cov` expects column numbers to be bytes, not code points #119033

Zalathar · 2023-12-17T01:10:25Z

Normally the compiler emits column numbers as a 1-based number of Unicode code points.

But when we embed coverage mappings for -Cinstrument-coverage, those mappings will ultimately be read by the llvm-cov tool. That tool assumes that column numbers are 1-based numbers of bytes, and relies on that assumption when slicing up source code to apply highlighting (in HTML reports, and in text-based reports with colour).

For the very common case of all-ASCII source code, bytes and code points are the same, so the difference isn't noticeable. But for code that contains non-ASCII characters, emitting column numbers as code points will result in llvm-cov slicing strings in the wrong places, producing mangled output or fatal errors.

(See taiki-e/cargo-llvm-cov#275 as an example of what can go wrong.)

Allow coverage tests to ignore test modes, and to enable color in coverage reports This PR adds two new header directives to compiletest, intended for use by coverage tests (and by rust-lang#119033 in particular). The new headers are: - `// ignore-mode-{mode}` causes a test to not be run in a particular compiletest mode (e.g. `ignore-mode-coverage-run`). - This can theoretically be used by any test, but coverage tests are currently the only ones that automatically run in multiple modes, so it's not very useful for other kinds of test. - `// llvm-cov-flags: --use-color` makes `coverage-run` tests pass the flag `--use-color` when generating coverage reports. - For most tests, non-coloured reports are easier to read and more portable across platforms. But for rust-lang#119033 specifically, we want to test that `llvm-cov` slices up source text correctly, which only happens when colour output is enabled.

Rollup merge of rust-lang#119034 - Zalathar:ignore-mode, r=davidtwco Allow coverage tests to ignore test modes, and to enable color in coverage reports This PR adds two new header directives to compiletest, intended for use by coverage tests (and by rust-lang#119033 in particular). The new headers are: - `// ignore-mode-{mode}` causes a test to not be run in a particular compiletest mode (e.g. `ignore-mode-coverage-run`). - This can theoretically be used by any test, but coverage tests are currently the only ones that automatically run in multiple modes, so it's not very useful for other kinds of test. - `// llvm-cov-flags: --use-color` makes `coverage-run` tests pass the flag `--use-color` when generating coverage reports. - For most tests, non-coloured reports are easier to read and more portable across platforms. But for rust-lang#119033 specifically, we want to test that `llvm-cov` slices up source text correctly, which only happens when colour output is enabled.

Zalathar · 2024-01-06T01:34:04Z

#119034 has landed, so now I can think about making this ready for review.

rustbot · 2024-01-06T01:50:06Z

Some changes occurred to MIR optimizations

cc @rust-lang/wg-mir-opt

rustbot · 2024-01-06T01:50:47Z

Error: Parsing assign command in comment failed: ...'' | error: specify user to assign to at >| ''...

Please file an issue on GitHub at triagebot if there's a problem with this bot, or reach out on #t-infra on Zulip.

Zalathar · 2024-01-06T01:51:17Z

r? compiler

compiler/rustc_mir_transform/src/coverage/mod.rs

davidtwco

r=me when CI passes

oli-obk · 2024-01-08T11:33:30Z

@bors delegate+

bors · 2024-01-08T11:33:33Z

✌️ @Zalathar, you can now approve this pull request!

If @oli-obk told you to "r=me" after making some further change, please make that change, then do @bors r=@oli-obk

Zalathar · 2024-01-08T11:52:44Z

@bors r=davidtwco

bors · 2024-01-08T11:52:47Z

📌 Commit 6971e93 has been approved by davidtwco

It is now in the queue for this repository.

coverage: `llvm-cov` expects column numbers to be bytes, not code points Normally the compiler emits column numbers as a 1-based number of Unicode code points. But when we embed coverage mappings for `-Cinstrument-coverage`, those mappings will ultimately be read by the `llvm-cov` tool. That tool assumes that column numbers are 1-based numbers of *bytes*, and relies on that assumption when slicing up source code to apply highlighting (in HTML reports, and in text-based reports with colour). For the very common case of all-ASCII source code, bytes and code points are the same, so the difference isn't noticeable. But for code that contains non-ASCII characters, emitting column numbers as code points will result in `llvm-cov` slicing strings in the wrong places, producing mangled output or fatal errors. (See taiki-e/cargo-llvm-cov#275 as an example of what can go wrong.)

…iaskrgr Rollup of 8 pull requests Successful merges: - rust-lang#118903 (Improved support of collapse_debuginfo attribute for macros.) - rust-lang#119033 (coverage: `llvm-cov` expects column numbers to be bytes, not code points) - rust-lang#119654 (bump bootstrap dependencies) - rust-lang#119660 (remove an unnecessary stderr-per-bitwidth) - rust-lang#119663 (tests: Normalize `\r\n` to `\n` in some run-make tests) - rust-lang#119681 (coverage: Anonymize line numbers in branch views) - rust-lang#119704 (Fix two variable binding issues in lint let_underscore) - rust-lang#119725 (Add helper for when we want to know if an item has a host param) r? `@ghost` `@rustbot` modify labels: rollup

coverage: `llvm-cov` expects column numbers to be bytes, not code points Normally the compiler emits column numbers as a 1-based number of Unicode code points. But when we embed coverage mappings for `-Cinstrument-coverage`, those mappings will ultimately be read by the `llvm-cov` tool. That tool assumes that column numbers are 1-based numbers of *bytes*, and relies on that assumption when slicing up source code to apply highlighting (in HTML reports, and in text-based reports with colour). For the very common case of all-ASCII source code, bytes and code points are the same, so the difference isn't noticeable. But for code that contains non-ASCII characters, emitting column numbers as code points will result in `llvm-cov` slicing strings in the wrong places, producing mangled output or fatal errors. (See taiki-e/cargo-llvm-cov#275 as an example of what can go wrong.)

…iaskrgr Rollup of 11 pull requests Successful merges: - rust-lang#116343 (Stop mentioning internal lang items in no_std binary errors) - rust-lang#118903 (Improved support of collapse_debuginfo attribute for macros.) - rust-lang#119033 (coverage: `llvm-cov` expects column numbers to be bytes, not code points) - rust-lang#119598 (Fix a typo in core::ops::Deref's doc) - rust-lang#119660 (remove an unnecessary stderr-per-bitwidth) - rust-lang#119663 (tests: Normalize `\r\n` to `\n` in some run-make tests) - rust-lang#119681 (coverage: Anonymize line numbers in branch views) - rust-lang#119704 (Fix two variable binding issues in lint let_underscore) - rust-lang#119725 (Add helper for when we want to know if an item has a host param) - rust-lang#119738 (Add `riscv32imafc-esp-espidf` tier 3 target for the ESP32-P4.) - rust-lang#119740 (Remove crossbeam-channel) Failed merges: - rust-lang#119723 (Remove `-Zdont-buffer-diagnostics`.) r? `@ghost` `@rustbot` modify labels: rollup

…iaskrgr Rollup of 10 pull requests Successful merges: - rust-lang#118903 (Improved support of collapse_debuginfo attribute for macros.) - rust-lang#119033 (coverage: `llvm-cov` expects column numbers to be bytes, not code points) - rust-lang#119598 (Fix a typo in core::ops::Deref's doc) - rust-lang#119660 (remove an unnecessary stderr-per-bitwidth) - rust-lang#119663 (tests: Normalize `\r\n` to `\n` in some run-make tests) - rust-lang#119681 (coverage: Anonymize line numbers in branch views) - rust-lang#119704 (Fix two variable binding issues in lint let_underscore) - rust-lang#119725 (Add helper for when we want to know if an item has a host param) - rust-lang#119738 (Add `riscv32imafc-esp-espidf` tier 3 target for the ESP32-P4.) - rust-lang#119740 (Remove crossbeam-channel) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#119033 - Zalathar:unicode, r=davidtwco coverage: `llvm-cov` expects column numbers to be bytes, not code points Normally the compiler emits column numbers as a 1-based number of Unicode code points. But when we embed coverage mappings for `-Cinstrument-coverage`, those mappings will ultimately be read by the `llvm-cov` tool. That tool assumes that column numbers are 1-based numbers of *bytes*, and relies on that assumption when slicing up source code to apply highlighting (in HTML reports, and in text-based reports with colour). For the very common case of all-ASCII source code, bytes and code points are the same, so the difference isn't noticeable. But for code that contains non-ASCII characters, emitting column numbers as code points will result in `llvm-cov` slicing strings in the wrong places, producing mangled output or fatal errors. (See taiki-e/cargo-llvm-cov#275 as an example of what can go wrong.)

Zalathar · 2024-03-01T11:56:08Z

@rustbot label +relnotes

This is a user-visible bug fix for taiki-e/cargo-llvm-cov#275.

Suggested summary: “Fix coverage instrumentation/reports for non-ASCII source code”

Pkgsrc changes: * Adapt checksums and patches. Upstream chnages: Version 1.77.0 (2024-03-21) ========================== - [Reveal opaque types within the defining body for exhaustiveness checking.] (rust-lang/rust#116821) - [Stabilize C-string literals.] (rust-lang/rust#117472) - [Stabilize THIR unsafeck.] (rust-lang/rust#117673) - [Add lint `static_mut_refs` to warn on references to mutable statics.] (rust-lang/rust#117556) - [Support async recursive calls (as long as they have indirection).] (rust-lang/rust#117703) - [Undeprecate lint `unstable_features` and make use of it in the compiler.] (rust-lang/rust#118639) - [Make inductive cycles in coherence ambiguous always.] (rust-lang/rust#118649) - [Get rid of type-driven traversal in const-eval interning] (rust-lang/rust#119044), only as a [future compatiblity lint] (rust-lang/rust#122204) for now. - [Deny braced macro invocations in let-else.] (rust-lang/rust#119062) Compiler -------- - [Include lint `soft_unstable` in future breakage reports.] (rust-lang/rust#116274) - [Make `i128` and `u128` 16-byte aligned on x86-based targets.] (rust-lang/rust#116672) - [Use `--verbose` in diagnostic output.] (rust-lang/rust#119129) - [Improve spacing between printed tokens.] (rust-lang/rust#120227) - [Merge the `unused_tuple_struct_fields` lint into `dead_code`.] (rust-lang/rust#118297) - [Error on incorrect implied bounds in well-formedness check] (rust-lang/rust#118553), with a temporary exception for Bevy. - [Fix coverage instrumentation/reports for non-ASCII source code.] (rust-lang/rust#119033) - [Fix `fn`/`const` items implied bounds and well-formedness check.] (rust-lang/rust#120019) - [Promote `riscv32{im|imafc}-unknown-none-elf` targets to tier 2.] (rust-lang/rust#118704) - Add several new tier 3 targets: - [`aarch64-unknown-illumos`] (rust-lang/rust#112936) - [`hexagon-unknown-none-elf`] (rust-lang/rust#117601) - [`riscv32imafc-esp-espidf`] (rust-lang/rust#119738) - [`riscv32im-risc0-zkvm-elf`] (rust-lang/rust#117958) Refer to Rust's [platform support page][platform-support-doc] for more information on Rust's tiered platform support. Libraries --------- - [Implement `From<&[T; N]>` for `Cow<[T]>`.] (rust-lang/rust#113489) - [Remove special-case handling of `vec.split_off (0)`.](rust-lang/rust#119917) Stabilized APIs --------------- - [`array::each_ref`] (https://doc.rust-lang.org/stable/std/primitive.array.html#method.each_ref) - [`array::each_mut`] (https://doc.rust-lang.org/stable/std/primitive.array.html#method.each_mut) - [`core::net`] (https://doc.rust-lang.org/stable/core/net/index.html) - [`f32::round_ties_even`] (https://doc.rust-lang.org/stable/std/primitive.f32.html#method.round_ties_even) - [`f64::round_ties_even`] (https://doc.rust-lang.org/stable/std/primitive.f64.html#method.round_ties_even) - [`mem::offset_of!`] (https://doc.rust-lang.org/stable/std/mem/macro.offset_of.html) - [`slice::first_chunk`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.first_chunk) - [`slice::first_chunk_mut`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.first_chunk_mut) - [`slice::split_first_chunk`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.split_first_chunk) - [`slice::split_first_chunk_mut`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.split_first_chunk_mut) - [`slice::last_chunk`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.last_chunk) - [`slice::last_chunk_mut`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.last_chunk_mut) - [`slice::split_last_chunk`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.split_last_chunk) - [`slice::split_last_chunk_mut`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.split_last_chunk_mut) - [`slice::chunk_by`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.chunk_by) - [`slice::chunk_by_mut`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.chunk_by_mut) - [`Bound::map`] (https://doc.rust-lang.org/stable/std/ops/enum.Bound.html#method.map) - [`File::create_new`] (https://doc.rust-lang.org/stable/std/fs/struct.File.html#method.create_new) - [`Mutex::clear_poison`] (https://doc.rust-lang.org/stable/std/sync/struct.Mutex.html#method.clear_poison) - [`RwLock::clear_poison`] (https://doc.rust-lang.org/stable/std/sync/struct.RwLock.html#method.clear_poison) Cargo ----- - [Extend the build directive syntax with `cargo::`.] (rust-lang/cargo#12201) - [Stabilize metadata `id` format as `PackageIDSpec`.] (rust-lang/cargo#12914) - [Pull out as `cargo-util-schemas` as a crate.] (rust-lang/cargo#13178) - [Strip all debuginfo when debuginfo is not requested.] (rust-lang/cargo#13257) - [Inherit jobserver from env for all kinds of runners.] (rust-lang/cargo#12776) - [Deprecate rustc plugin support in cargo.] (rust-lang/cargo#13248) Rustdoc ----- - [Allows links in markdown headings.] (rust-lang/rust#117662) - [Search for tuples and unit by type with `()`.] (rust-lang/rust#118194) - [Clean up the source sidebar's hide button.] (rust-lang/rust#119066) - [Prevent JS injection from `localStorage`.] (rust-lang/rust#120250) Misc ---- - [Recommend version-sorting for all sorting in style guide.] (rust-lang/rust#115046) Internal Changes ---------------- These changes do not affect any public interfaces of Rust, but they represent significant improvements to the performance or internals of rustc and related tools. - [Add more weirdness to `weird-exprs.rs`.] (rust-lang/rust#119028)

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. A-code-coverage Area: Source-based code coverage (-Cinstrument-coverage) labels Dec 17, 2023

Zalathar mentioned this pull request Dec 17, 2023

Allow coverage tests to ignore test modes, and to enable color in coverage reports #119034

Merged

Swatinem approved these changes Dec 17, 2023

View reviewed changes

This comment was marked as resolved.

Sign in to view

Zalathar force-pushed the unicode branch from ae031ef to f14cf71 Compare December 28, 2023 03:20

Zalathar force-pushed the unicode branch from f14cf71 to 349a9c1 Compare January 6, 2024 01:32

Zalathar marked this pull request as ready for review January 6, 2024 01:50

This comment was marked as resolved.

Sign in to view

rustbot assigned davidtwco Jan 6, 2024

Zalathar mentioned this pull request Jan 6, 2024

coverage: Never emit improperly-ordered coverage regions #119460

Merged

oli-obk reviewed Jan 8, 2024

View reviewed changes

compiler/rustc_mir_transform/src/coverage/mod.rs Outdated Show resolved Hide resolved

Zalathar added 2 commits January 8, 2024 21:43

coverage: Test for column numbers involving non-ASCII characters

585a285

coverage: Allow make_code_region to fail

88f5759

Zalathar force-pushed the unicode branch from 349a9c1 to 4060418 Compare January 8, 2024 10:49

coverage: llvm-cov expects column numbers to be bytes, not code points

6971e93

Zalathar force-pushed the unicode branch from 4060418 to 6971e93 Compare January 8, 2024 10:58

davidtwco approved these changes Jan 8, 2024

View reviewed changes

bors removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jan 8, 2024

bors added the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Jan 8, 2024

matthiaskrgr mentioned this pull request Jan 8, 2024

Rollup of 8 pull requests #119737

Closed

matthiaskrgr mentioned this pull request Jan 8, 2024

Rollup of 8 pull requests #119745

Closed

matthiaskrgr mentioned this pull request Jan 8, 2024

Rollup of 11 pull requests #119747

Closed

matthiaskrgr mentioned this pull request Jan 8, 2024

Rollup of 10 pull requests #119754

Merged

bors merged commit 70e3f8d into rust-lang:master Jan 9, 2024
11 checks passed

rustbot added this to the 1.77.0 milestone Jan 9, 2024

Zalathar deleted the unicode branch January 9, 2024 02:24

Zalathar mentioned this pull request Jan 9, 2024

html output break utf-8 taiki-e/cargo-llvm-cov#275

Closed

rustbot added the relnotes Marks issues that should be documented in the release notes of the next release. label Mar 1, 2024

Zalathar mentioned this pull request Mar 17, 2024

coverage: Replace color terminal tests with HTML output tests #122631

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

coverage: `llvm-cov` expects column numbers to be bytes, not code points #119033

coverage: `llvm-cov` expects column numbers to be bytes, not code points #119033

Zalathar commented Dec 17, 2023 •

edited

This comment was marked as resolved.

Zalathar commented Jan 6, 2024

rustbot commented Jan 6, 2024

This comment was marked as resolved.

rustbot commented Jan 6, 2024

Zalathar commented Jan 6, 2024

davidtwco left a comment

oli-obk commented Jan 8, 2024

bors commented Jan 8, 2024

Zalathar commented Jan 8, 2024

bors commented Jan 8, 2024

Zalathar commented Mar 1, 2024

coverage: llvm-cov expects column numbers to be bytes, not code points #119033

coverage: llvm-cov expects column numbers to be bytes, not code points #119033

Conversation

Zalathar commented Dec 17, 2023 • edited

This comment was marked as resolved.

Zalathar commented Jan 6, 2024

rustbot commented Jan 6, 2024

This comment was marked as resolved.

rustbot commented Jan 6, 2024

Zalathar commented Jan 6, 2024

davidtwco left a comment

Choose a reason for hiding this comment

oli-obk commented Jan 8, 2024

bors commented Jan 8, 2024

Zalathar commented Jan 8, 2024

bors commented Jan 8, 2024

Zalathar commented Mar 1, 2024

coverage: `llvm-cov` expects column numbers to be bytes, not code points #119033

coverage: `llvm-cov` expects column numbers to be bytes, not code points #119033

Zalathar commented Dec 17, 2023 •

edited