Use `token::Lit` in `ast::ExprKind::Lit`. #102944

nnethercote · 2022-10-12T04:09:45Z

Instead of ast::Lit.

Literal lowering now happens at two different times. Expression literals are lowered when HIR is crated. Attribute literals are lowered during parsing.

r? @petrochenkov

rustbot · 2022-10-12T04:09:48Z

Some changes occurred in src/tools/clippy

cc @rust-lang/clippy

nnethercote · 2022-10-12T04:11:54Z

Another alternative to the literal changes in #101562 and #101885.

I don't feel entirely happy with this approach. Various things:

The output change to src/test/ui/lint/lint-impl-fn.rs is bogus, but I don't understand why it happens.
rustc_ast_lowering now depends on rustc_parse, for report_lit_error, because lit lowering now happens in both of those crates.
ast::Lit still has redundancy, which two literal "kinds" within it.
There are some almost duplicate paths in the parser:
- parse_opt_token_lit / parse_opt_ast_lit
- parse_token_lit / parse_ast_lit`

nnethercote · 2022-10-12T04:12:26Z

Also, it's not yet complete. I haven't updated clippy and rustfmt. I'll do that if we decide that this is the right path forward.

petrochenkov · 2022-10-13T11:11:36Z

rustc_ast_lowering now depends on rustc_parse, for report_lit_error, because lit lowering now happens in both of those crates.

The literal lowering logic, including error reporting, is independent of the rest of the parser and can be moved to some common dependency of AST->HIR lowering and Attribute -> MetaItem lowering, or passed through a function pointer like nt_to_tokenstream before #97251.

ast::Lit still has redundancy, which two literal "kinds" within it.

I'd expect it to be renamed to something like MetaItemLit, and the token_lit part to be removed (not in this PR).

petrochenkov · 2022-10-13T11:41:42Z

Change description for the lang team.

This PR changes some errors related to literals from syntactic-but-not-in-macros to semantic.
These are the error names in the compiler:

    InvalidSuffix,
    InvalidIntSuffix,
    InvalidFloatSuffix,
    NonDecimalFloat(u32),
    IntTooLarge,

Examples:

"string"any_suffix
10u123
10.0f123
0b10f32
999340282366920938463463374607431768211455999

All these examples are valid tokens, that can be passed to macros without errors, but previously they were reported as errors when parsing proper Rust code, including code that will be later removed by cfg or attribute macros.

// Before
sink! {
    "string"any_suffix // OK
    10u123 // OK
    10.0f123 // OK
    0b10f32 // OK
    999340282366920938463463374607431768211455999 // OK
}

#[cfg(FALSE)]
fn configured_out() {
    "string"any_suffix // ERROR
    10u123 // ERROR
    10.0f123 // ERROR
    0b10f32 // ERROR
    999340282366920938463463374607431768211455999 // ERROR
}

fn main() {
    "string"any_suffix // ERROR
    10u123 // ERROR
    10.0f123 // ERROR
    0b10f32 // ERROR
    999340282366920938463463374607431768211455999 // ERROR
}

This PR makes them fully semantic errors so they are only reported after all macros are expanded.

// After
sink! {
    "string"any_suffix // OK
    10u123 // OK
    10.0f123 // OK
    0b10f32 // OK
    999340282366920938463463374607431768211455999 // OK
}

#[cfg(FALSE)]
fn configured_out() {
    "string"any_suffix // OK
    10u123 // OK
    10.0f123 // OK
    0b10f32 // OK
    999340282366920938463463374607431768211455999 // OK
}

fn main() {
    "string"any_suffix // ERROR
    10u123 // ERROR
    10.0f123 // ERROR
    0b10f32 // ERROR
    999340282366920938463463374607431768211455999 // ERROR
}

This way

we make behavior of function-like and attribute-like macros more consistent
we make other literals more consistent with floating point literals which are already converted from strings to floats lately in the compiler (probably due to expensiveness of the conversion)
we avoid unnecessary work in the compiler - literals don't need to be checked/converted unless they are really used, and size of expression AST nodes can be shrank.

nnethercote · 2022-10-14T01:09:21Z

we make other literals more consistent with floating point literals which are already converted from strings to floats lately in the compiler (probably due to expensiveness of the conversion)

Floats are stored as Symbols rather than f64 in ast::LitKind so that ast::LitKind can implement Eq and Hash. I think this is the real reason they are converted later, rather than cost.

y86-dev · 2022-10-18T19:04:03Z

0x10.0 // OK

This is currently not accepted by the compiler: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=316edfc43c4ad955ee7b667883a10287

petrochenkov · 2022-10-18T19:17:25Z

@y86-dev
Ah, you are right, 0x10.0 -> 0b10f32 in all the examples.
UPD: I've updated the examples in #102944 (comment).

nnethercote · 2022-10-20T21:40:37Z

@scottmcm: The meeting notes indicate some support for this, but also that a firm conclusion wasn't reached. Should we do an FCP?

scottmcm · 2022-10-23T06:35:50Z

@nnethercote I've been trying to distill my thoughts here into something less fuzzy. I think I see three different situations here.

Things that match currently-existing token forms, but are hitting arbitrary length restrictions.

For example, decimal literals that are obviously lexically fine, like

#[cfg(nope)]
pub fn foo() {
    let x = 1111111111222222222233333333334444444444;
}

For that kind, I think we had lang consensus that they definitely shouldn't be a tokenizing nor parsing time error, but rather a semantic error later. So making those work for macros and cfg'd-out code as I understand this PR would do makes great sense to me.

Extensions of existing lexical forms to other widths.

I don't think we're likely to add u1024 any time soon, but if we ever did I think it's very obvious that 0_u1024 would be a valid value of the type, as would 0xFFFF_u1024 and the other obvious examples. Similarly for 0.0_f16, or other such things with different bitwidths. (0.0_f11 is massively unlikely, but I still don't think it's something we'd need to block.)

I think lang also felt positive about supporting these, though I don't think we talked about them as much. Personally I'm definitely in favour.

Things where the combination has no meaning today.

This is where I'm not so confident about things. When it explicitly rejects hexadecimal and binary floating point, it's not at all obvious to me that 0b0f32 is something we want to support for situations like this. What does that even mean, especially when 0x0f32 does something completely related to what an f32 suffix means near a .? Similarly, it's not obvious to me at all that "hello"match should even be one token. I guess we can't break that in macros until an edition, so it's stuck being passable there, but that doesn't mean it's something that makes sense to me to allow through the parser.

Hopefully that's understandable 🤞

How do you feel about those distinctions? Is the line between cases 1&2 and case 3 something that you think is meaningful? How obnoxious would it be to separate those cases in the code? Am I missing something about how Rust is set up lexically/syntactically?

cc @rust-lang/lang for comments in hopes this makes more sense than my rambling in the meeting.

nnethercote · 2022-10-24T22:12:45Z

I have no strong opinions about the language design issues here. I'm just interested in shrinking the size of ast::Literal, in order to shrink ast::Expr, in order to reduce compile times.

@petrochenkov: what do think about Scott's comments?

petrochenkov · 2022-10-25T09:27:14Z

@scottmcm

How do you feel about those distinctions? Is the line between cases 1&2 and case 3 something that you think is meaningful?

I wouldn't make any of these distinctions.
For me the important part is that all of them are considered valid tokens (accepted as tt or proc macro input), so it's too late to make the distinctions and they should all be accepted by Rust parser as well.
If you want to make distinctions, then perhaps the moral of this story is that we need to pay more attention when stabilizing something as a valid token.

How obnoxious would it be to separate those cases in the code? Am I missing something about how Rust is set up lexically/syntactically?

"string"any_suffix - less obnoxious, the suffix check is cheap and AST expressions can still keep token::Lit, although it may be annoying to not report this error twice.
everything else - more obnoxious, AST either needs to keep lowered literals, or the compiler will have to lower literals twice doing more work instead of less work.

joshtriplett · 2022-10-25T18:40:48Z

We discussed this in today's @rust-lang/lang meeting, and we decided that we're willing to evaluate this via FCP. We're not thrilled that these were handled in the first place, but given that they work in macros, we should let them work in cfg(FALSE). That doesn't stop us from giving them language-level meanings in the parser later.

joshtriplett · 2022-10-25T18:40:55Z

@rfcbot merge

rfcbot · 2022-10-25T18:40:56Z

Team member @joshtriplett has proposed to merge this. The next step is review by the rest of the tagged team members:

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

Instead of `ast::Lit`. Literal lowering now happens at two different times. Expression literals are lowered when HIR is crated. Attribute literals are lowered during parsing. This commit changes the language very slightly. Some programs that used to not compile now will compile. This is because some invalid literals that are removed by `cfg` or attribute macros will no longer trigger errors. See this comment for more details: rust-lang/rust#102944 (comment)

This reflects the changes made in in rust-lang/rust#102944 . Previously, unknown suffixes were rejected by the parser. Now they are accepted by the parser and rejected at a later stage. Similarly, integer literals too large to fit in u128 are now accepted by the parser. Forms like 5f32 are now INTEGER_LITERAL rather than FLOAT_LITERAL. The notion of a 'pseudoliteral' is no longer required.

…, r=petrochenkov Use `token::Lit` in `ast::ExprKind::Lit`. Instead of `ast::Lit`. Literal lowering now happens at two different times. Expression literals are lowered when HIR is crated. Attribute literals are lowered during parsing. r? `@petrochenkov`

rylev · 2022-11-22T14:36:04Z

Note from perf triage: the regressions are largely concentrated in primary benchmarks while the improvements are in secondary. So while the regressions and improvements balance out when treating all benchmarks equally, I do think that primary regressions should be considered to have more weight than secondary improvements.

Update literal suffix docs for rust-lang/rust#102944

Update books ## rust-lang/book 1 commits in 3f64052c048c6def93b94a2b514ee88bba918744..a60f4316ec923a5ac2ed6a2eba6960edb832d855 2022-11-16 15:07:18 UTC to 2022-11-16 15:07:18 UTC - Fix Install MdBook command (rust-lang/book#3424) ## rust-embedded/book 4 commits in c533348edd69f11a8f4225d633a05d7093fddbf3..19f798d448835a4888e3b3eae7fe69f1d61d8681 2022-11-17 15:08:11 UTC to 2022-11-08 23:27:57 UTC - start/hardware.md: Fix typo (rust-embedded/book#336) - doc: Fix `arm-none-eabi-gdb` installation instruction for Fedora 27 or newer to just use `gdb` (rust-embedded/book#335) - Update singletons.md (rust-embedded/book#334) - Remove incorrect claim HashMap is avail in no_std (rust-embedded/book#333) ## rust-lang/nomicon 2 commits in 05532356e7a4dbea2330aabb77611f5179493bb8..ae406aa5287a9e025abb72343aaceec98458c117 2022-11-21 22:48:20 UTC to 2022-11-15 00:29:20 UTC - Improve chapter about `Vec<T>` (rust-lang/nomicon#381) - Grammar change for 3.4: Limits of Lifetimes (lifetime-mismatch.md) (rust-lang/nomicon#386) ## rust-lang/reference 9 commits in 9f0cc13ffcd27c1fbe1ab766a9491e15ddcf4d19..3ae62681ff236d5528ef7c8c28ba7c6b2ecc6731 2022-12-05 00:51:50 UTC to 2022-11-15 20:43:30 UTC - Document that type parameter `Self` is unsized by default (rust-lang/reference#1285) - replace `crateid` term with `crate_name` (rust-lang/reference#1310) - Document native library modifier `verbatim` (rust-lang/reference#1299) - Update literal suffix docs for rust-lang#102944 (rust-lang/reference#1305) - update aliasing rules section of the reference (rust-lang/reference#1290) - Document RFC 2867: instruction_set attribute (rust-lang/reference#1253) - Fix a minor typo in the "Higher-ranked trait bounds" section (rust-lang/reference#1288) - Disallow newline directly following `//` (rust-lang/reference#1294) - Add an anchor to the "forwarding macro fragments" paragraph (rust-lang/reference#1300) ## rust-lang/rust-by-example 5 commits in 2b15c0abf2bada6e00553814336bc3e2d8399097..a9869b4a3c4cac3bc6099b41f088679e268400b8 2022-11-27 19:03:05 UTC to 2022-11-11 18:54:53 UTC - Migrate from highfive to triagebot (rust-lang/rust-by-example#1647) - Simpler version of the read_lines script. (rust-lang/rust-by-example#1641) - Fix comment in "Formatted print" example code (rust-lang/rust-by-example#1638) - Added a missing backtick in a comment in chapter 15.4. (rust-lang/rust-by-example#1642) - Clarify the confusing closure example rust-lang#1611 (rust-lang/rust-by-example#1643) ## rust-lang/rustc-dev-guide 13 commits in d0dc6c97a6486f68bac782fff135086eae6d77ec..e269950a57fa6fcda356426545fb5aa3691a7ced 2022-12-03 23:09:24 UTC to 2022-11-08 21:35:38 UTC - Remove duplicate paragraph (rust-lang/rustc-dev-guide#1523) - clarify subtree tool policy (rust-lang/rustc-dev-guide#1518) - Typo (rust-lang/rustc-dev-guide#1520) - Link directly to the section on `--keep-stage` (rust-lang/rustc-dev-guide#1515) - do an actual link to detect if it breaks in future (rust-lang/rustc-dev-guide#1517) - Triage some date-check items (rust-lang/rustc-dev-guide#1513) - Update path for `try_mark_green` implementation (rust-lang/rustc-dev-guide#1512) - Fix a broken design docs link about unused substs bug (rust-lang/rustc-dev-guide#1511) - updating-llvm: keep a calm tone (rust-lang/rustc-dev-guide#1449) - date-check: updating-llvm (rust-lang/rustc-dev-guide#1424) - rewrite the section about regions in the trait solver (rust-lang/rustc-dev-guide#1508) - Consistent ordered list indexing (rust-lang/rustc-dev-guide#1509) - Document multiple alternative suggestions on diagnostic structs (rust-lang/rustc-dev-guide#1486)

…, r=petrochenkov Use `token::Lit` in `ast::ExprKind::Lit`. Instead of `ast::Lit`. Literal lowering now happens at two different times. Expression literals are lowered when HIR is crated. Attribute literals are lowered during parsing. r? `@petrochenkov`

Instead of `ast::Lit`. Literal lowering now happens at two different times. Expression literals are lowered when HIR is crated. Attribute literals are lowered during parsing. This commit changes the language very slightly. Some programs that used to not compile now will compile. This is because some invalid literals that are removed by `cfg` or attribute macros will no longer trigger errors. See this comment for more details: rust-lang/rust#102944 (comment)

Pkgsrc changes: * Adjust patches and cargo checksums to new versions, but also one strange "mips" conditional. Upstream changes: Version 1.67.0 (2023-01-26) ========================== Language -------- - [Make `Sized` predicates coinductive, allowing cycles.] (rust-lang/rust#100386) - [`#[must_use]` annotations on `async fn` also affect the `Future::Output`.] (rust-lang/rust#100633) - [Elaborate supertrait obligations when deducing closure signatures.] (rust-lang/rust#101834) - [Invalid literals are no longer an error under `cfg(FALSE)`.] (rust-lang/rust#102944) - [Unreserve braced enum variants in value namespace.] (rust-lang/rust#103578) Compiler -------- - [Enable varargs support for calling conventions other than `C` or `cdecl`.] (rust-lang/rust#97971) - [Add new MIR constant propagation based on dataflow analysis.] (rust-lang/rust#101168) - [Optimize field ordering by grouping m\*2^n-sized fields with equivalently aligned ones.] (rust-lang/rust#102750) - [Stabilize native library modifier `verbatim`.] (rust-lang/rust#104360) Added and removed targets: - [Add a tier 3 target for PowerPC on AIX] (rust-lang/rust#102293), `powerpc64-ibm-aix`. - [Add a tier 3 target for the Sony PlayStation 1] (rust-lang/rust#102689), `mipsel-sony-psx`. - [Add tier 3 `no_std` targets for the QNX Neutrino RTOS] (rust-lang/rust#102701), `aarch64-unknown-nto-qnx710` and `x86_64-pc-nto-qnx710`. - [Remove tier 3 `linuxkernel` targets] (rust-lang/rust#104015) (not used by the actual kernel). Refer to Rust's [platform support page][platform-support-doc] for more information on Rust's tiered platform support. Libraries --------- - [Merge `crossbeam-channel` into `std::sync::mpsc`.] (rust-lang/rust#93563) - [Fix inconsistent rounding of 0.5 when formatted to 0 decimal places.] (rust-lang/rust#102935) - [Derive `Eq` and `Hash` for `ControlFlow`.] (rust-lang/rust#103084) - [Don't build `compiler_builtins` with `-C panic=abort`.] (rust-lang/rust#103786) Stabilized APIs --------------- - [`{integer}::checked_ilog`] (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.checked_ilog) - [`{integer}::checked_ilog2`] (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.checked_ilog2) - [`{integer}::checked_ilog10`] (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.checked_ilog10) - [`{integer}::ilog`] (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.ilog) - [`{integer}::ilog2`] (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.ilog2) - [`{integer}::ilog10`] (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.ilog10) - [`NonZeroU*::ilog2`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroU32.html#method.ilog2) - [`NonZeroU*::ilog10`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroU32.html#method.ilog10) - [`NonZero*::BITS`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroU32.html#associatedconstant.BITS) These APIs are now stable in const contexts: - [`char::from_u32`] (https://doc.rust-lang.org/stable/std/primitive.char.html#method.from_u32) - [`char::from_digit`] (https://doc.rust-lang.org/stable/std/primitive.char.html#method.from_digit) - [`char::to_digit`] (https://doc.rust-lang.org/stable/std/primitive.char.html#method.to_digit) - [`core::char::from_u32`] (https://doc.rust-lang.org/stable/core/char/fn.from_u32.html) - [`core::char::from_digit`] (https://doc.rust-lang.org/stable/core/char/fn.from_digit.html) Compatibility Notes ------------------- - [The layout of `repr(Rust)` types now groups m\*2^n-sized fields with equivalently aligned ones.] (rust-lang/rust#102750) This is intended to be an optimization, but it is also known to increase type sizes in a few cases for the placement of enum tags. As a reminder, the layout of `repr(Rust)` types is an implementation detail, subject to change. - [0.5 now rounds to 0 when formatted to 0 decimal places.] (rust-lang/rust#102935) This makes it consistent with the rest of floating point formatting that rounds ties toward even digits. - [Chains of `&&` and `||` will now drop temporaries from their sub-expressions in evaluation order, left-to-right.] (rust-lang/rust#103293) Previously, it was "twisted" such that the _first_ expression dropped its temporaries _last_, after all of the other expressions dropped in order. - [Underscore suffixes on string literals are now a hard error.] (rust-lang/rust#103914) This has been a future-compatibility warning since 1.20.0. - [Stop passing `-export-dynamic` to `wasm-ld`.] (rust-lang/rust#105405) - [`main` is now mangled as `__main_void` on `wasm32-wasi`.] (rust-lang/rust#105468) - [Cargo now emits an error if there are multiple registries in the configuration with the same index URL.] (rust-lang/cargo#10592) Internal Changes ---------------- These changes do not affect any public interfaces of Rust, but they represent significant improvements to the performance or internals of rustc and related tools. - [Rewrite LLVM's archive writer in Rust.] (rust-lang/rust#97485)

Pkgsrc changes: * Adjust patches (add & remove) and cargo checksums to new versions. * It's conceivable that the workaround for LLVM based NetBSD works even less in this version (ref. PKGSRC_HAVE_LIBCPP not having a corresponding patch anymore). Upstream changes: Version 1.68.2 (2023-03-28) =========================== - [Update the GitHub RSA host key bundled within Cargo] (rust-lang/cargo#11883). The key was [rotated by GitHub] (https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/) on 2023-03-24 after the old one leaked. - [Mark the old GitHub RSA host key as revoked] (rust-lang/cargo#11889). This will prevent Cargo from accepting the leaked key even when trusted by the system. - [Add support for `@revoked` and a better error message for `@cert-authority` in Cargo's SSH host key verification] (rust-lang/cargo#11635) Version 1.68.1 (2023-03-23) =========================== - [Fix miscompilation in produced Windows MSVC artifacts] (rust-lang/rust#109094) This was introduced by enabling ThinLTO for the distributed rustc which led to miscompilations in the resulting binary. Currently this is believed to be limited to the -Zdylib-lto flag used for rustc compilation, rather than a general bug in ThinLTO, so only rustc artifacts should be affected. - [Fix --enable-local-rust builds] (rust-lang/rust#109111) - [Treat `$prefix-clang` as `clang` in linker detection code] (rust-lang/rust#109156) - [Fix panic in compiler code] (rust-lang/rust#108162) Version 1.68.0 (2023-03-09) =========================== Language -------- - [Stabilize default_alloc_error_handler] (rust-lang/rust#102318) This allows usage of `alloc` on stable without requiring the definition of a handler for allocation failure. Defining custom handlers is still unstable. - [Stabilize `efiapi` calling convention.] (rust-lang/rust#105795) - [Remove implicit promotion for types with drop glue] (rust-lang/rust#105085) Compiler -------- - [Change `bindings_with_variant_name` to deny-by-default] (rust-lang/rust#104154) - [Allow .. to be parsed as let initializer] (rust-lang/rust#105701) - [Add `armv7-sony-vita-newlibeabihf` as a tier 3 target] (rust-lang/rust#105712) - [Always check alignment during compile-time const evaluation] (rust-lang/rust#104616) - [Disable "split dwarf inlining" by default.] (rust-lang/rust#106709) - [Add vendor to Fuchsia's target triple] (rust-lang/rust#106429) - [Enable sanitizers for s390x-linux] (rust-lang/rust#107127) Libraries --------- - [Loosen the bound on the Debug implementation of Weak.] (rust-lang/rust#90291) - [Make `std::task::Context` !Send and !Sync] (rust-lang/rust#95985) - [PhantomData layout guarantees] (rust-lang/rust#104081) - [Don't derive Debug for `OnceWith` & `RepeatWith`] (rust-lang/rust#104163) - [Implement DerefMut for PathBuf] (rust-lang/rust#105018) - [Add O(1) `Vec -> VecDeque` conversion guarantee] (rust-lang/rust#105128) - [Leak amplification for peek_mut() to ensure BinaryHeap's invariant is always met] (rust-lang/rust#105851) Stabilized APIs --------------- - [`{core,std}::pin::pin!`] (https://doc.rust-lang.org/stable/std/pin/macro.pin.html) - [`impl From<bool> for {f32,f64}`] (https://doc.rust-lang.org/stable/std/primitive.f32.html#impl-From%3Cbool%3E-for-f32) - [`std::path::MAIN_SEPARATOR_STR`] (https://doc.rust-lang.org/stable/std/path/constant.MAIN_SEPARATOR_STR.html) - [`impl DerefMut for PathBuf`] (https://doc.rust-lang.org/stable/std/path/struct.PathBuf.html#impl-DerefMut-for-PathBuf) These APIs are now stable in const contexts: - [`VecDeque::new`] (https://doc.rust-lang.org/stable/std/collections/struct.VecDeque.html#method.new) Cargo ----- - [Stabilize sparse registry support for crates.io] (rust-lang/cargo#11224) - [`cargo build --verbose` tells you more about why it recompiles.] (rust-lang/cargo#11407) - [Show progress of crates.io index update even `net.git-fetch-with-cli` option enabled] (rust-lang/cargo#11579) Misc ---- Compatibility Notes ------------------- - [Add `SEMICOLON_IN_EXPRESSIONS_FROM_MACROS` to future-incompat report] (rust-lang/rust#103418) - [Only specify `--target` by default for `-Zgcc-ld=lld` on wasm] (rust-lang/rust#101792) - [Bump `IMPLIED_BOUNDS_ENTAILMENT` to Deny + ReportNow] (rust-lang/rust#106465) - [`std::task::Context` no longer implements Send and Sync] (rust-lang/rust#95985) nternal Changes ---------------- These changes do not affect any public interfaces of Rust, but they represent significant improvements to the performance or internals of rustc and related tools. - [Encode spans relative to the enclosing item] (rust-lang/rust#84762) - [Don't normalize in AstConv] (rust-lang/rust#101947) - [Find the right lower bound region in the scenario of partial order relations] (rust-lang/rust#104765) - [Fix impl block in const expr] (rust-lang/rust#104889) - [Check ADT fields for copy implementations considering regions] (rust-lang/rust#105102) - [rustdoc: simplify JS search routine by not messing with lev distance] (rust-lang/rust#105796) - [Enable ThinLTO for rustc on `x86_64-pc-windows-msvc`] (rust-lang/rust#103591) - [Enable ThinLTO for rustc on `x86_64-apple-darwin`] (rust-lang/rust#103647) Version 1.67.0 (2023-01-26) ========================== Language -------- - [Make `Sized` predicates coinductive, allowing cycles.] (rust-lang/rust#100386) - [`#[must_use]` annotations on `async fn` also affect the `Future::Output`.] (rust-lang/rust#100633) - [Elaborate supertrait obligations when deducing closure signatures.] (rust-lang/rust#101834) - [Invalid literals are no longer an error under `cfg(FALSE)`.] (rust-lang/rust#102944) - [Unreserve braced enum variants in value namespace.] (rust-lang/rust#103578) Compiler -------- - [Enable varargs support for calling conventions other than `C` or `cdecl`.] (rust-lang/rust#97971) - [Add new MIR constant propagation based on dataflow analysis.] (rust-lang/rust#101168) - [Optimize field ordering by grouping m\*2^n-sized fields with equivalently aligned ones.] (rust-lang/rust#102750) - [Stabilize native library modifier `verbatim`.] (rust-lang/rust#104360) Added and removed targets: - [Add a tier 3 target for PowerPC on AIX] (rust-lang/rust#102293), `powerpc64-ibm-aix`. - [Add a tier 3 target for the Sony PlayStation 1] (rust-lang/rust#102689), `mipsel-sony-psx`. - [Add tier 3 `no_std` targets for the QNX Neutrino RTOS] (rust-lang/rust#102701), `aarch64-unknown-nto-qnx710` and `x86_64-pc-nto-qnx710`. - [Remove tier 3 `linuxkernel` targets] (rust-lang/rust#104015) (not used by the actual kernel). Refer to Rust's [platform support page][platform-support-doc] for more information on Rust's tiered platform support. Libraries --------- - [Merge `crossbeam-channel` into `std::sync::mpsc`.] (rust-lang/rust#93563) - [Fix inconsistent rounding of 0.5 when formatted to 0 decimal places.] (rust-lang/rust#102935) - [Derive `Eq` and `Hash` for `ControlFlow`.] (rust-lang/rust#103084) - [Don't build `compiler_builtins` with `-C panic=abort`.] (rust-lang/rust#103786) Stabilized APIs --------------- - [`{integer}::checked_ilog`] (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.checked_ilog) - [`{integer}::checked_ilog2`] (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.checked_ilog2) - [`{integer}::checked_ilog10`] (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.checked_ilog10) - [`{integer}::ilog`] (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.ilog) - [`{integer}::ilog2`] (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.ilog2) - [`{integer}::ilog10`] (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.ilog10) - [`NonZeroU*::ilog2`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroU32.html#method.ilog2) - [`NonZeroU*::ilog10`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroU32.html#method.ilog10) - [`NonZero*::BITS`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroU32.html#associatedconstant.BITS) These APIs are now stable in const contexts: - [`char::from_u32`] (https://doc.rust-lang.org/stable/std/primitive.char.html#method.from_u32) - [`char::from_digit`] (https://doc.rust-lang.org/stable/std/primitive.char.html#method.from_digit) - [`char::to_digit`] (https://doc.rust-lang.org/stable/std/primitive.char.html#method.to_digit) - [`core::char::from_u32`] (https://doc.rust-lang.org/stable/core/char/fn.from_u32.html) - [`core::char::from_digit`] (https://doc.rust-lang.org/stable/core/char/fn.from_digit.html) Compatibility Notes ------------------- - [The layout of `repr(Rust)` types now groups m\*2^n-sized fields with equivalently aligned ones.] (rust-lang/rust#102750) This is intended to be an optimization, but it is also known to increase type sizes in a few cases for the placement of enum tags. As a reminder, the layout of `repr(Rust)` types is an implementation detail, subject to change. - [0.5 now rounds to 0 when formatted to 0 decimal places.] (rust-lang/rust#102935) This makes it consistent with the rest of floating point formatting that rounds ties toward even digits. - [Chains of `&&` and `||` will now drop temporaries from their sub-expressions in evaluation order, left-to-right.] (rust-lang/rust#103293) Previously, it was "twisted" such that the _first_ expression dropped its temporaries _last_, after all of the other expressions dropped in order. - [Underscore suffixes on string literals are now a hard error.] (rust-lang/rust#103914) This has been a future-compatibility warning since 1.20.0. - [Stop passing `-export-dynamic` to `wasm-ld`.] (rust-lang/rust#105405) - [`main` is now mangled as `__main_void` on `wasm32-wasi`.] (rust-lang/rust#105468) - [Cargo now emits an error if there are multiple registries in the configuration with the same index URL.] (rust-lang/cargo#10592) Internal Changes ---------------- These changes do not affect any public interfaces of Rust, but they represent significant improvements to the performance or internals of rustc and related tools. - [Rewrite LLVM's archive writer in Rust.] (rust-lang/rust#97485)

Currently string literals are unescaped twice. - Once during lexing in `cook_quoted`/`cook_c_string`/`cook_common`. This one just checks for errors. - Again in `LitKind::from_token_lit`, which is mostly called when lowering AST to HIR, but also in a few other places during expansion. This one actually constructs the unescaped string. It also has error checking code, but that code handling the error cases is actually dead (and has several bugs) because the check during lexing catches all errors! This commit removes the checking during lexing, and fixes up `LitKind::from_token_lit` so it properly does both checking and construction. This is a language change: some programs now compile that previously did not. For example, it is now possible for macros to be passed "invalid" string literals like "\a\b\c". This is a continuation of a trend of delaying semantic error checking of literals to after expansion, e.g. rust-lang#102944 did this for some cases for numeric literals, and the detection of NUL chars in C string literals is already delayed in this way. XXX: have Session::report_lit_errors? XXX: have LitKind::from_token_lit so you don't need the .0? Things to note: - `LitError` has a new `EscapeError` variant. - `LitKind::from_token_lit`'s return value changed, to produce multiple errors/warnings, and also to handle lexer warnings. This latter case is annoying but necessary to preserve existing warning behaviour. - `report_lit_error` becomes `report_lit_errors`, in order to handle multiple errors in a single string literal. Notes about test changes: - `tests/rustdoc-ui/ignore-block-help.rs`: this relies on a parsing error occurring. The error present was an unescaping error, which is now delayed to after parsing. So the commit changes it to an "unterminated character literal" error which continues to occurs during parsing. - Several tests had unescaping errors combined with unterminated literal errors. The former are now delayed but the latter remain as lexing errors. So the unterminated literal part needed to be split into a separate test file otherwise compilation would end before the other errors were reported. - issue-62913.rs: The structure and output changed a bit. Issue rust-lang#62913 was about an ICE due to an unterminated string literal, so the new version should be good enough. - literals-are-validated-before-expansion.rs: this tests exactly the behaviour that has been changed, and so was removed XXX: insert a new test covering more of that - A couple of other test produce the same errors, just in a different order.

XXX: need more tests Currently string literals are unescaped twice. - Once during lexing in `cook_quoted`/`cook_c_string`/`cook_common`. This one just checks for errors. - Again in `LitKind::from_token_lit`, which is called when lowering AST to HIR, and also in a few other places during expansion. This one actually constructs the unescaped string. It also has error checking code, but that part of the code is actually dead (and has several bugs) because the check during lexing catches all errors! This commit removes the error-check-only unescaping during lexing, and fixes up `LitKind::from_token_lit` so it properly does both checking and construction. This is a backwards-compatible language change: some programs now compile that previously did not. For example, it is now possible for macros to consume "invalid" string literals like "\a\b\c". This is a continuation of a trend of delaying semantic error checking of literals to after expansion: - rust-lang#102944 did this for some cases for numeric literals - The detection of NUL chars in C string literals is already delayed in this way. Notes about test changes: - `tests/rustdoc-ui/ignore-block-help.rs`: this requires a parse error occurring. The error used was an unescaping error, which is now delayed to after parsing. So the commit changes it to an "unterminated character literal" error which still occurs during parsing. - Several tests had unescaping errors combined with unterminated literal errors. The former are now delayed but the latter remain as lexing errors. So the unterminated literal part needed to be split into a separate test file otherwise compilation would end before the other errors were reported. - issue-62913.rs: The structure and output changed a bit. Issue rust-lang#62913 was about an ICE due to an unterminated string literal, so the new version should be good enough. - literals-are-validated-before-expansion.rs: this tests exactly the behaviour that has been changed, and so was removed - A couple of other test produce the same errors, just in a different order.

Currently string literals are unescaped twice. - Once during lexing in `cook_quoted`/`cook_c_string`/`cook_common`. This one just checks for errors. - Again in `LitKind::from_token_lit`, which is called when lowering AST to HIR, and also in a few other places during expansion. This one actually constructs the unescaped string. It also has error checking code, but that part of the code is actually dead (and has several bugs) because the check during lexing catches all errors! This commit removes the error-check-only unescaping during lexing, and fixes up `LitKind::from_token_lit` so it properly does both checking and construction. This is a backwards-compatible language change: some programs now compile that previously did not. For example, it is now possible for macros to consume "invalid" string literals like "\a\b\c". This is a continuation of a trend of delaying semantic error checking of literals to after expansion: - rust-lang#102944 did this for some cases for numeric literals - The detection of NUL chars in C string literals is already delayed in this way. Notes about test changes: - `ignore-block-help.rs`: this requires a parse error for the test to work. The error used was an unescaping error, which is now delayed to after parsing. So the commit changes it to an "unterminated character literal" error which still occurs during parsing. - `tests/ui/lexer/error-stage.rs`: this shows the newly allowed cases, due to delayed literal unescaping. - Several tests had unescaping errors combined with unterminated literal errors. The former are now delayed but the latter remain as lexing errors. So the unterminated literal part needed to be split into a separate test file otherwise compilation would end before the other errors were reported. - issue-62913.rs: The structure and output changed a bit. Issue rust-lang#62913 was about an ICE due to an unterminated string literal, so the new version should be good enough. - literals-are-validated-before-expansion.rs: this tests exactly the behaviour that has been changed, and so was removed - A couple of other test produce the same errors, just in a different order.

rustbot added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Oct 12, 2022

rust-highfive assigned petrochenkov Oct 12, 2022

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 12, 2022

This comment has been minimized.

Sign in to view

This comment was marked as resolved.

Sign in to view

rfcbot added proposed-final-comment-period Proposed to merge/close by relevant subteam, see T-<team> label. Will enter FCP once signed off. disposition-merge This issue / PR is in PFCP or FCP with a disposition to merge it. labels Oct 25, 2022

joshtriplett removed the I-lang-nominated The issue / PR has been nominated for discussion during a lang team meeting. label Oct 25, 2022

rfcbot added final-comment-period In the final comment period and will be merged soon unless new substantive objections are raised. and removed proposed-final-comment-period Proposed to merge/close by relevant subteam, see T-<team> label. Will enter FCP once signed off. labels Oct 26, 2022

jruderman mentioned this pull request Nov 19, 2022

ICE "token_lit in from_mac_args: InvalidIntSuffix" #104620

Closed

mattheww mentioned this pull request Nov 20, 2022

Update literal suffix docs for rust-lang/rust#102944 rust-lang/reference#1305

Merged

nnethercote mentioned this pull request Nov 20, 2022

Remove ast::Lit::kind #101885

Closed

jruderman mentioned this pull request Nov 23, 2022

ICE: unreachable match arm in concat_bytes::invalid_type_err #104769

Closed

ehuss added a commit to rust-lang/reference that referenced this pull request Nov 25, 2022

Merge pull request #1305 from mattheww/2022-11_parse_all_suffixes

c7a39ca

Update literal suffix docs for rust-lang/rust#102944

ehuss mentioned this pull request Nov 27, 2022

Update books #104993

Closed

jruderman mentioned this pull request Nov 28, 2022

ICE when invalid string literal used with macro #105011

Closed

rustbot mentioned this pull request Dec 5, 2022

Update books #105313

Merged

apiraino removed the to-announce Announce this issue on triage meeting label Jan 5, 2023

est31 mentioned this pull request Mar 10, 2023

Implement RFC 3348, c"foo" literals #108801

Merged

nnethercote mentioned this pull request Dec 12, 2023

Delay literal unescaping #118699

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use `token::Lit` in `ast::ExprKind::Lit`. #102944

Use `token::Lit` in `ast::ExprKind::Lit`. #102944

nnethercote commented Oct 12, 2022

rustbot commented Oct 12, 2022

nnethercote commented Oct 12, 2022

nnethercote commented Oct 12, 2022

This comment has been minimized.

petrochenkov commented Oct 13, 2022

petrochenkov commented Oct 13, 2022 •

edited

nnethercote commented Oct 14, 2022

y86-dev commented Oct 18, 2022

petrochenkov commented Oct 18, 2022 •

edited

nnethercote commented Oct 20, 2022

This comment was marked as resolved.

scottmcm commented Oct 23, 2022

nnethercote commented Oct 24, 2022

petrochenkov commented Oct 25, 2022

joshtriplett commented Oct 25, 2022

joshtriplett commented Oct 25, 2022

rfcbot commented Oct 25, 2022 •

edited by cramertj

rylev commented Nov 22, 2022

Use token::Lit in ast::ExprKind::Lit. #102944

Use token::Lit in ast::ExprKind::Lit. #102944

Conversation

nnethercote commented Oct 12, 2022

rustbot commented Oct 12, 2022

nnethercote commented Oct 12, 2022

nnethercote commented Oct 12, 2022

This comment has been minimized.

petrochenkov commented Oct 13, 2022

petrochenkov commented Oct 13, 2022 • edited

nnethercote commented Oct 14, 2022

y86-dev commented Oct 18, 2022

petrochenkov commented Oct 18, 2022 • edited

nnethercote commented Oct 20, 2022

This comment was marked as resolved.

scottmcm commented Oct 23, 2022

nnethercote commented Oct 24, 2022

petrochenkov commented Oct 25, 2022

joshtriplett commented Oct 25, 2022

joshtriplett commented Oct 25, 2022

rfcbot commented Oct 25, 2022 • edited by cramertj

rylev commented Nov 22, 2022

Use `token::Lit` in `ast::ExprKind::Lit`. #102944

Use `token::Lit` in `ast::ExprKind::Lit`. #102944

petrochenkov commented Oct 13, 2022 •

edited

petrochenkov commented Oct 18, 2022 •

edited

rfcbot commented Oct 25, 2022 •

edited by cramertj