Stabilize count, ignore, index, and length in Rust 1.80

[c410-f3r]

Stabilization proposal

This PR proposes the stabilization of a subset of #![feature(macro_metavar_expr)] or more specifically, the stabilization of count, ignore, index and length.

Tracking issue: #83527
Version: 1.80 (June 13 2024 => beta, July 25 2024 => stable).

What is stabilized

Count

The number of times a meta variable repeats in total.

The output of count depends on where it is placed as well the provided index. If no index is provided, then it will always start at the innermost level.

macro_rules! count_idents {
    ( $( $i:ident ),* ) => {
        ${count($i)}
    };
}

fn main() {
    assert_eq!(count_idents!(a, b, c), 3);
}

Ignore

Binds a meta variable for repetition, but expands to nothing.

macro_rules! count {
    ( $( $i:stmt ),* ) => {{
        0 $( + 1 ${ignore($i)} )*
    }};
}

fn main() {
    assert_eq!(count!(if true {} else {}, let _: () = (), || false), 3);
}

Index

The current index of the inner-most repetition.

index is a counter that starts from 0 until the end of the repetition.

trait Foo {
    fn bar(&self) -> usize;
}

macro_rules! impl_tuple {
    ( $( $name:ident ),* ) => {
        impl<$( $name, )*> Foo for ($( $name, )*)
        where
            $( $name: AsRef<[u8]>, )*
        {
            fn bar(&self) -> usize {
                let mut sum: usize = 0;
                $({
                    const $name: () = ();
                    sum = sum.wrapping_add(self.${index()}.as_ref().len());
                })*
                sum
            }
        }
    };
}

impl_tuple!(A, B, C, D);

fn main() {
}

Length

The current index starting from the inner-most repetition.

length indicates the sum of meta variable elements, aka length, of the current repetition.

macro_rules! array_3d {
    ( $( $( $number:literal ),* );* ) => {
        [
            $(
                [
                    $( $number + ${length()}, )*
                ],
            )*
        ]
    };
}

fn main() {
    assert_eq!(array_3d!(0, 1; 2, 3; 4, 5), [[2, 3], [4, 5], [6, 7]]);
}

Motivation

Meta variable expressions not only facilitate the use of macros but also allow things that can't be done today like in the $index example.

An initial effort to stabilize this feature was made in #111908 but ultimately reverted because of possible obstacles related to syntax and expansion.

Nevertheless, #83527 (comment) tried to address some questions and fortunately the lang team accept #117050 the unblocking suggestions.

Here we are today after ~4 months so everything should be mature enough for wider use.

What isn't stabilized

$$ is not being stabilized due to unresolved concerns.

History

• 2021-02-22, RFC: Declarative macro metavariable expressions
• 2021-03-26, Tracking Issue for RFC 3086: macro metavariable expressions
• 2022-02-01, Implement macro meta-variable expressions
• 2022-02-25, [1/2] Implement macro meta-variable expressions
• 2022-03-11, [2/2] Implement macro meta-variable expressions
• 2022-03-12, Fix remaining meta-variable expression TODOs
• 2019-03-21, [macro-metavar-expr] Fix generated tokens hygiene
• 2022-04-07, Kickstart the inner usage of macro_metavar_expr
• 2022-04-07, [macro_metavar_expr] Add tests to ensure the feature requirement
• 2023-05-24, [RFC-3086] Restrict the parsing of count
• 2023-10-22, [RFC 3086] Attempt to try to resolve blocking concerns

Tests

This list is a subset of https://github.com/rust-lang/rust/tree/master/tests/ui/macros/rfc-3086-metavar-expr.

• Ensures that nested macros have correct behavior

• Compares produced tokens to assert expected outputs

• Checks the declarations of the features

• Verifies all possible errors that can occur due to incorrect user input

Possible future work

With enough consensus, other nightly expressions can be added for experimentation and possibly stabilized in the future. For example, #118958.

Thanks @markbt for creating the RFC and thanks to @petrochenkov and @mark-i-m for reviewing the implementations.

[rustbot]

r? @fmease

rustbot has assigned @fmease.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[fmease]

@c410-f3r Has this been documented in the Reference already? The box is not ticked over at the tracking issue. If not, stabilization is blocked on adding sufficient documentation.

[joshtriplett]

@rfcbot merge

@rfcbot concern document-in-reference

[rfcbot]

Team member @joshtriplett has proposed to merge this. The next step is review by the rest of the tagged team members:

• @joshtriplett
• @nikomatsakis
• @pnkfelix
• @scottmcm
• @tmandry

Concerns:

• document-in-reference (Stabilize count, ignore, index, and length in Rust 1.80 #122808 (comment))
• hygiene-handled? resolved by Stabilize count, ignore, index, and length in Rust 1.80 #122808 (comment)

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.

[RalfJung]

Length

The current index starting from the inner-most repetition.

What does this mean, and why does "length" not show up in the explanation of the length "function"? I would expect this to be the length of the inner-most repetition, is that not the case?

All these "functions" expand to integers -- of which type? Or do they end up being like integer literals without a suffix, i.e., the concrete integer type gets inferred following the usual rules?

[Veykril]

Note that the tracking issue still has an unresolved question/bug open #83527

Figure out problems around hygiene

Has this been resolved?

[joshtriplett]

@rfcbot concern hygiene-handled?

[Veykril]

Ah actually, it seems the checkbox was never ticked? #95188

[joshtriplett]

@rfcbot resolved hygiene-handled?

[lcnr]

what is the difference between index and length?

[tgross35]

Are the functions that take depth such as ${count(ident, depth)} included as part of this?

Nonblocker, diagnostics are pretty bare bones with no error codes or help

error: `count` can not be placed inside the inner-most repetition
 --> src/main.rs:7:19
  |
7 |             dbg!(${count($i, 0)});
  |                   ^^^^^^^^^^^^^^

[slanterns]

• 2019-03-21, [macro-metavar-expr] Fix generated tokens hygiene

Date seems incorrect :)

[nagisa]

To follow up on the question on the depth of count, does "in total" mean this considers all levels of repetition? Like, if I write$( $a: ident: $($b: literal),* );+ and this matches a a: 1, 2, 3; b: 4, 5, 6, do I get count($b) of 3 twice, or a count($b) of 6 twice? Or does it depend on where I write the count at?

The RFC text does not really make this clear…

[c410-f3r]

@fmease

Has this been documented in the Reference already?

No but documentation will be provided as soon as possible.

@RalfJung

What does this mean, and why does "length" not show up in the explanation of the length "function"? I would expect this to be the length of the inner-most repetition, is that not the case?

length indicates the sum of meta variable elements, aka length, of the current repetition and that is why, contrary the count, it can only be placed inside repetitions.

The nested repetitions of https://user-images.githubusercontent.com/17877264/158820060-035b2548-9ea9-4cbb-850b-e5836cf892fe.png illustrate better this concept.

More information is available at https://rust-lang.github.io/rfcs/3086-macro-metavar-expr.html#index-and-length

All these "functions" expand to integers -- of which type? Or do they end up being like integer literals without a suffix, i.e., the concrete integer type gets inferred following the usual rules?

length and count are expanded to unsuffixed integer literals.

More information is available at https://rust-lang.github.io/rfcs/3086-macro-metavar-expr.html#count

rust/compiler/rustc_expand/src/mbe/transcribe.rs

Lines 677 to 683 in df8ac8f

	MetaVarExpr::Length(depth) => match repeats.iter().nth_back(depth) {
	Some((_, length)) => {
	result.push(TokenTree::token_alone(
	TokenKind::lit(token::Integer, sym::integer(*length), None),
	visited_span(),
	));
	}

@lcnr

what is the difference between index and length?

length indicates the sum of meta variable elements of the current repetition while index is a counter that starts from 0 until the end of the repetition.

macro_rules! index {
    ( $( $tt:tt ),* ) => {
        [$( ${ignore($tt)} ${index()}, )*]
    };
}

macro_rules! length {
    ( $( $tt:tt ),* ) => {
        [$( ${ignore($tt)} ${length()}, )*]
    };
}

fn main() {
    // 3 tokens generate an increment list of 3 elements starting from 0
    assert_eq!(index!(A, B, C), [0, 1, 2]);
    
    // 4 tokens generate a list of 4 elements containing 4
    assert_eq!(length!(A, B, C, D), [4, 4, 4, 4]);
}

More information is available at https://rust-lang.github.io/rfcs/3086-macro-metavar-expr.html#index-and-length

@ slanterns

Date seems incorrect :)

Thanks

@nagisa

To follow up on the question on the depth of count, does "in total" mean this considers all levels of repetition? Like, if I write $( $a: ident: $($b: literal),* );+ and this matches a a: 1, 2, 3; b: 4, 5, 6, do I get count($b) of 3 twice, or a count($b) of 6 twice? Or does it depend on where I write the count at?

count depends on where it is placed as well the provided index. If no index is provided, then it will always start at the innermost level.

So in your example a declaration of $( $a:ident: $($b: literal),* );+ called with a: 1, 2, 3; b: 4, 5, 6, will produce 3, 3 if called inside the outermost repetition or 6 if called without any repetition.

// Other examples

macro_rules! no_repetition {
    ( $( $a:ident: $( $b:literal ),* );+ ) => {
        [${count($b)}]
    };
}

macro_rules! no_repetition_with_index {
    ( $( $a:ident: $( $b:literal ),* );+ ) => {
        [${count($b, 1)}]
    };
}

macro_rules! outermost_repetition {
    ( $( $a:ident: $( $b:literal ),* );+ ) => {
        [$( ${ignore($a)} ${count($b)}, )+]
    };
}

fn main() {
    // 1 2 3 4 5 = 5 elements
    assert_eq!(no_repetition!(a: 1, 2, 3; b: 4, 5), [5]);
    
    // a b = 2 elements
    assert_eq!(no_repetition_with_index!(a: 1, 2, 3; b: 4, 5), [2]);
    
    // 1 2 3 = 3 elements, 4 5 = 2 elements
    assert_eq!(outermost_repetition!(a: 1, 2, 3; b: 4, 5), [3, 2]);
}

https://user-images.githubusercontent.com/17877264/158820060-035b2548-9ea9-4cbb-850b-e5836cf892fe.png might help you but it was created in a time when count started at the outermost level.

* PR updated

[jdahlstrom]

So does ${count()} equal ${length()} in the most common case of having only a single level of repetition?

[danielhenrymantilla]

@jdahlstrom
IIUC, I believe:

[$( $x / ${length()} ),*]

to be morally equivalent to:

let n = ${count(x)};
[$( $x / n ),*]

• (notice where the operator lies w.r.t. the $()* repetition)

[tgross35]

Would it be possible to prepare the PR for the reference before this lands? It seems like there is quite a bit of uncertainty about how all the features work, which makes me thing that perhaps this feature isn't very widely tested. As semantics have changed a few times since the RFC, that isn't the best reference either.

Quick search for usage that isn't in rustc forks https://github.com/search?q=lang%3Arust+%2F%28%3F-i%29%5C%24%5C%7B%28ignore%7Ccount%7Clength%7Cindex%29%5C%28.*%5C%7D%2F+-path%3Acompiler%2F+-path%3Alibrary%2F+-path%3Asrc%2Ftools%2Frust-analyzer&type=code

[c410-f3r]

Has this been documented in the Reference already?

No but documentation will be provided as soon as possible.

As commented above, reference will be provided as soon as possible.

Semantic changed only once when count moved from outermost to innermost. #117050 also required the addition of $ but this is more syntax-related than an actual change of how the feature works, everything else remains the same as when the RFC was approved 2 years ago.

[workingjubilee]

Is there any realistic pattern to suffix the integer literal expansions or nah?

[programmerjake]

Is there any realistic pattern to suffix the integer literal expansions or nah?

well, I expect ${concat(${length()}, _u32)} to work at some point...

[bors]

☔ The latest upstream changes (presumably #123390) made this pull request unmergeable. Please resolve the merge conflicts.

[nikomatsakis]

@rfcbot reviewed -- this would definitely be handy!

[jdahlstrom]

I brought this up in the doc PR but it belongs here – length should probably be renamed len before stabilization. The latter is de facto standard in the standard library, whereas the former is only used in a single unstable API. These metafunctions aren’t library items of course, but should presumably still be consistent with established names.

[workingjubilee]

I'm slightly worried about random inference regressions punishing this particularly hard if there's no standard/easy way to proof the emitted literals against such. I guess we can use a block expression for each one?

[bors]

☔ The latest upstream changes (presumably #104087) made this pull request unmergeable. Please resolve the merge conflicts.