format_args can create incorrect subspans for raw string literals that adopted another string literal’s span – assertion failed: bpos.to_u32() >= mbc.pos.to_u32() + mbc.bytes as u32

[steffahn]

Playing around with indoc, I’ve noticed that while a format string that indoc doesn’t modify further, like

indoc::printdoc!(
"x{}"
);

leads to an error message that has a span pointing into the string literal

error: 1 positional argument in format string, but no arguments were given
 --> src/main.rs:8:3
  |
8 | "x{}"
  |   ^^

When the string is modified by indoc (removing some leading whitespace) like

indoc::printdoc!(
"
 x{}"
);

that leads to an error message spanning the whole string literal.

error: 1 positional argument in format string, but no arguments were given
 --> src/main.rs:8:1
  |
8 | / "
9 | |  x{}"
  | |_____^

So far so good, this all looks intentional. However with raw string literals, the behavior is no longer the same:

Without modification

indoc::printdoc!(
r"x{}"
);

error: 1 positional argument in format string, but no arguments were given
 --> src/main.rs:8:4
  |
8 | r"x{}"
  |    ^^

and with modification by indoc

indoc::printdoc!(
r"
 x{}"
);

error: 1 positional argument in format string, but no arguments were given
 --> src/main.rs:9:1
  |
9 |  x{}"
  | ^^

Now we have a span pointing to the wrong location.

---

Apparently that’s sufficient to make the compiler crash, if multi-byte characters are involved. E.g.

indoc::printdoc!(
r"
 字{}"
);

thread 'rustc' panicked at 'assertion failed: bpos.to_u32() >= mbc.pos.to_u32() + mbc.bytes as u32', compiler/rustc_span/src/lib.rs:1754:17
stack backtrace:
   0:     0x7f7d50b31821 - std::backtrace_rs::backtrace::libunwind::trace::hac6b0a2f611a19eb
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x7f7d50b31821 - std::backtrace_rs::backtrace::trace_unsynchronized::h5781e26c6497eba1
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7f7d50b31821 - std::sys_common::backtrace::_print_fmt::hdc1fb874acdfa993
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x7f7d50b31821 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h53605d02a89bf381
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x7f7d50b92b2f - core::fmt::rt::Argument::fmt::h5cd3573cfe525dec
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/core/src/fmt/rt.rs:138:9
   5:     0x7f7d50b92b2f - core::fmt::write::h8e8c77da1d63560a
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/core/src/fmt/mod.rs:1094:21
   6:     0x7f7d50b24ad1 - std::io::Write::write_fmt::h65d39cb1760e375f
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/std/src/io/mod.rs:1713:15
   7:     0x7f7d50b31635 - std::sys_common::backtrace::_print::hbc9d82330e0ee61b
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x7f7d50b31635 - std::sys_common::backtrace::print::h3df96419e2ece626
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x7f7d50b34317 - std::panicking::default_hook::{{closure}}::h42ca76a1a85e81e6
  10:     0x7f7d50b34104 - std::panicking::default_hook::h305e9ec0e2c1602a
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/std/src/panicking.rs:288:9
  11:     0x7f7d53d6bcdb - <rustc_driver_impl[137076bf347c04e0]::install_ice_hook::{closure#0} as core[d52b761e2ec85dac]::ops::function::FnOnce<(&core[d52b761e2ec85dac]::panic::panic_info::PanicInfo,)>>::call_once::{shim:vtable#0}
  12:     0x7f7d50b34a5d - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h482c4de0d3d09ca7
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/alloc/src/boxed.rs:1999:9
  13:     0x7f7d50b34a5d - std::panicking::rust_panic_with_hook::h3cf33647be67d87f
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/std/src/panicking.rs:709:13
  14:     0x7f7d50b347b1 - std::panicking::begin_panic_handler::{{closure}}::h1972e04c78257d38
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/std/src/panicking.rs:595:13
  15:     0x7f7d50b31c56 - std::sys_common::backtrace::__rust_end_short_backtrace::he1e3ce014bb29c4b
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/std/src/sys_common/backtrace.rs:151:18
  16:     0x7f7d50b34542 - rust_begin_unwind
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/std/src/panicking.rs:593:5
  17:     0x7f7d50b8edb3 - core::panicking::panic_fmt::he055d12513132fe0
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/core/src/panicking.rs:67:14
  18:     0x7f7d50b8ee43 - core::panicking::panic::h0daaa0e8e6c70dc3
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/core/src/panicking.rs:117:5
  19:     0x7f7d5259a188 - <rustc_span[c2ff1acad35eadff]::source_map::SourceMap>::lookup_char_pos
  20:     0x7f7d5325ed7d - <rustc_errors[1d14d0959e896e80]::emitter::FileWithAnnotatedLines>::collect_annotations
  21:     0x7f7d5325052a - <rustc_errors[1d14d0959e896e80]::emitter::EmitterWriter>::emit_message_default::{closure#0}
  22:     0x7f7d5324b2d2 - <rustc_errors[1d14d0959e896e80]::emitter::EmitterWriter>::emit_messages_default
  23:     0x7f7d53242a8a - <rustc_errors[1d14d0959e896e80]::emitter::EmitterWriter as rustc_errors[1d14d0959e896e80]::emitter::Emitter>::emit_diagnostic
  24:     0x7f7d53241604 - <rustc_errors[1d14d0959e896e80]::json::Diagnostic>::from_errors_diagnostic
  25:     0x7f7d5323ef6c - <rustc_errors[1d14d0959e896e80]::json::JsonEmitter as rustc_errors[1d14d0959e896e80]::emitter::Emitter>::emit_diagnostic
  26:     0x7f7d534cf708 - <rustc_errors[1d14d0959e896e80]::HandlerInner>::emit_diagnostic::{closure#2}
  27:     0x7f7d534bfaa3 - rustc_interface[6e051652e0542a3f]::callbacks::track_diagnostic
  28:     0x7f7d52baa2f2 - <rustc_errors[1d14d0959e896e80]::HandlerInner>::emit_diagnostic
  29:     0x7f7d52b61c2e - <rustc_errors[1d14d0959e896e80]::Handler>::emit_diagnostic
  30:     0x7f7d52759aee - <rustc_span[c2ff1acad35eadff]::ErrorGuaranteed as rustc_errors[1d14d0959e896e80]::diagnostic_builder::EmissionGuarantee>::diagnostic_builder_emit_producing_guarantee
  31:     0x7f7d531c281d - rustc_builtin_macros[de0620bb92ddfcbb]::format::make_format_args
  32:     0x7f7d531bc82b - rustc_builtin_macros[de0620bb92ddfcbb]::format::expand_format_args_impl
  33:     0x7f7d52752eb6 - <rustc_expand[6f8d3c67003a1aaa]::expand::MacroExpander>::fully_expand_fragment
  34:     0x7f7d52e8d8b7 - <rustc_expand[6f8d3c67003a1aaa]::expand::MacroExpander>::expand_crate
  35:     0x7f7d52e8ccb0 - <rustc_session[9506f38856f586db]::session::Session>::time::<rustc_ast[540cd37e9df5b959]::ast::Crate, rustc_interface[6e051652e0542a3f]::passes::configure_and_expand::{closure#1}>
  36:     0x7f7d52e372e7 - rustc_interface[6e051652e0542a3f]::passes::resolver_for_lowering
  37:     0x7f7d531e524a - rustc_query_impl[49b48132f4839f96]::plumbing::__rust_begin_short_backtrace::<rustc_query_impl[49b48132f4839f96]::query_impl::resolver_for_lowering::dynamic_query::{closure#2}::{closure#0}, rustc_middle[40c22daa444bfe72]::query::erase::Erased<[u8; 8usize]>>
  38:     0x7f7d531e5239 - <rustc_query_impl[49b48132f4839f96]::query_impl::resolver_for_lowering::dynamic_query::{closure#2} as core[d52b761e2ec85dac]::ops::function::FnOnce<(rustc_middle[40c22daa444bfe72]::ty::context::TyCtxt, ())>>::call_once
  39:     0x7f7d53505e4d - rustc_query_system[9fe5c04c7931819b]::query::plumbing::try_execute_query::<rustc_query_impl[49b48132f4839f96]::DynamicConfig<rustc_query_system[9fe5c04c7931819b]::query::caches::SingleCache<rustc_middle[40c22daa444bfe72]::query::erase::Erased<[u8; 8usize]>>, false, false, false>, rustc_query_impl[49b48132f4839f96]::plumbing::QueryCtxt, true>
  40:     0x7f7d536ee90e - rustc_query_impl[49b48132f4839f96]::query_impl::resolver_for_lowering::get_query_incr::__rust_end_short_backtrace
  41:     0x7f7d53297083 - <rustc_middle[40c22daa444bfe72]::ty::context::GlobalCtxt>::enter::<rustc_driver_impl[137076bf347c04e0]::run_compiler::{closure#1}::{closure#2}::{closure#2}, &rustc_data_structures[a0fd66bc6a8ec75b]::steal::Steal<(rustc_middle[40c22daa444bfe72]::ty::ResolverAstLowering, alloc[e49b8bff3c7e14a6]::rc::Rc<rustc_ast[540cd37e9df5b959]::ast::Crate>)>>
  42:     0x7f7d53295dac - rustc_span[c2ff1acad35eadff]::set_source_map::<core[d52b761e2ec85dac]::result::Result<(), rustc_span[c2ff1acad35eadff]::ErrorGuaranteed>, rustc_interface[6e051652e0542a3f]::interface::run_compiler<core[d52b761e2ec85dac]::result::Result<(), rustc_span[c2ff1acad35eadff]::ErrorGuaranteed>, rustc_driver_impl[137076bf347c04e0]::run_compiler::{closure#1}>::{closure#0}::{closure#0}>
  43:     0x7f7d5328e0ff - std[51582e9e78a3fe4]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[6e051652e0542a3f]::util::run_in_thread_pool_with_globals<rustc_interface[6e051652e0542a3f]::interface::run_compiler<core[d52b761e2ec85dac]::result::Result<(), rustc_span[c2ff1acad35eadff]::ErrorGuaranteed>, rustc_driver_impl[137076bf347c04e0]::run_compiler::{closure#1}>::{closure#0}, core[d52b761e2ec85dac]::result::Result<(), rustc_span[c2ff1acad35eadff]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[d52b761e2ec85dac]::result::Result<(), rustc_span[c2ff1acad35eadff]::ErrorGuaranteed>>
  44:     0x7f7d5328da55 - <<std[51582e9e78a3fe4]::thread::Builder>::spawn_unchecked_<rustc_interface[6e051652e0542a3f]::util::run_in_thread_pool_with_globals<rustc_interface[6e051652e0542a3f]::interface::run_compiler<core[d52b761e2ec85dac]::result::Result<(), rustc_span[c2ff1acad35eadff]::ErrorGuaranteed>, rustc_driver_impl[137076bf347c04e0]::run_compiler::{closure#1}>::{closure#0}, core[d52b761e2ec85dac]::result::Result<(), rustc_span[c2ff1acad35eadff]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[d52b761e2ec85dac]::result::Result<(), rustc_span[c2ff1acad35eadff]::ErrorGuaranteed>>::{closure#1} as core[d52b761e2ec85dac]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  45:     0x7f7d50b3ef55 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h2c3489a66f80a759
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/alloc/src/boxed.rs:1985:9
  46:     0x7f7d50b3ef55 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc23ab4efd07db838
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/alloc/src/boxed.rs:1985:9
  47:     0x7f7d50b3ef55 - std::sys::unix::thread::Thread::new::thread_start::ha482bd14f5daf08a
                               at /rustc/6bba061467f7c2cab04b262b95eb67bf89265587/library/std/src/sys/unix/thread.rs:108:17
  48:     0x7f7d50a08ea7 - start_thread
  49:     0x7f7d50926a2f - clone
  50:                0x0 - <unknown>

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.72.0-nightly (6bba06146 2023-06-16) running on x86_64-unknown-linux-gnu

note: compiler flags: --crate-type bin -C embed-bitcode=no -C debuginfo=2 -C incremental=[REDACTED]

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [resolver_for_lowering] getting the resolver for lowering
end of query stack

(run on rustexplorer.com)

This is the assertion that fails:

rust/compiler/rustc_span/src/lib.rs

Lines 1760 to 1762 in c57393e

	// We should never see a byte position in the middle of a
	// character.
	assert!(bpos.to_u32() >= mbc.pos.to_u32() + mbc.bytes as u32);

---

As for a minimal reproduction, define a proc-macro

use proc_macro::{Delimiter, Group, Ident, Literal, Punct, Spacing, Span, TokenStream, TokenTree};

#[proc_macro]
pub fn foo(x: TokenStream) -> TokenStream {
    let span_tok = x.into_iter().next().unwrap();
    let mut lit: Literal = r#"r"{}""#.parse().unwrap();
    lit.set_span(span_tok.span());
    FromIterator::<TokenTree>::from_iter([
        Ident::new("println", Span::mixed_site()).into(),
        Punct::new('!', Spacing::Alone).into(),
        Group::new(Delimiter::Parenthesis, TokenTree::from(lit).into()).into(),
    ])
}

and then use it like so

use …::foo;
fn main() {
    foo!("字");
}

Another fun effect: If the macro-generated string is longer, e.g.

    let mut lit: Literal = r#"r"                      {}""#.parse().unwrap();

then the error message first points in other places in my program, and eventually also into different files:

error: 1 positional argument in format string, but no arguments were given
 --> /home/…USER…/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/lib.rs:1:9
  |
1 | //! # The Rust Standard Library
  |              ^^

This reproduces for latest stable and nightly.

---

> rustc +nightly --version
rustc 1.73.0-nightly (180dffba1 2023-08-14)
> rustc +stable --version 
rustc 1.71.1 (eb26296b5 2023-08-03)

[gurry]

@rustbot claim