futures_util::select! runs into recursion limits

[bryanburgers]

Whenever I use futures_util::select! from 0.3.0-alpha.19, I frequently run into the error

recursion limit reached while expanding the macro $crate::dispatch

It seems like if I have too many match arms within the select! { ... } macro, regardless of whether they are top-level or not, I get this error.

This means that when I have 4 different futures I'm selecting on, I can't really use match arms on most of them otherwise I get this error.

I can get around this by abstracting the logic into a different function and doing the logic there, which in general is a better approach anyway. But there are times when I want to use some match arms with a break, and it makes the code harder to read when I abstract the code into functions.

Below are a variety of different examples of how a certain amount of complexity causes the recursion error. Note that it seems like it's somehow a total of complexity, not any certain type.

This gives me the impression that somehow the macro is recursing into the portion of the code that it shouldn't be getting into? But I don't know enough about macros at this point to be able to diagnose.

fn main() {
    use futures_util::{select, stream::{self, StreamExt}, FutureExt};

    let future = async {
        let mut s1 = stream::repeat(0_usize);
        let mut s2 = stream::repeat(0_usize);
        let mut s3 = stream::repeat(0_usize);
        let mut s4 = stream::repeat(0_usize);
        let mut s5 = stream::repeat(0_usize);

        loop {
            /* Single stream with lots of match arms */
            select! {
                value = s1.next().fuse() => {
                    match value {
                        Some(0) => {},
                        Some(1) => {},
                        Some(2) => {},
                        Some(3) => {},
                        Some(4) => {},
                        Some(5) => {},
                        Some(6) => {},
                        Some(7) => {},
                        Some(8) => {},
                        Some(9) => {},
                        Some(10) => {},
                        Some(11) => {},
                        Some(12) => {},
                        Some(13) => {},
                        Some(14) => {},
                        Some(15) => {},
                        Some(16) => {},
                        Some(17) => {},
                        _ => {},
                    }
                },
            }

            /* Handful of streams with a some match arms */
            select! {
                value = s1.next().fuse() => {
                    match value {
                        Some(0) => {},
                        Some(1) => {},
                        Some(2) => {},
                        Some(3) => {},
                        _ => {},
                    }
                },
                value = s2.next().fuse() => {
                    match value {
                        Some(0) => {},
                        Some(1) => {},
                        Some(2) => {},
                        _ => {},
                    }
                },
                value = s3.next().fuse() => {
                    match value {
                        Some(0) => {},
                        Some(1) => {},
                        Some(2) => {},
                        Some(3) => {},
                        _ => {},
                    }
                },
            }

            /* Fair bit of streams with a few match arms */
            select! {
                value = s1.next().fuse() => {
                    match value {
                        Some(0) => {},
                        _ => {},
                    }
                },
                value = s2.next().fuse() => {
                    match value {
                        Some(0) => {},
                        _ => {},
                    }
                },
                value = s3.next().fuse() => {
                    match value {
                        Some(0) => {},
                        _ => {},
                    }
                },
                value = s4.next().fuse() => {
                    match value {
                        Some(0) => {},
                        _ => {},
                    }
                },
                value = s5.next().fuse() => {
                    match value {
                        Some(0) => {},
                        _ => {},
                    }
                },
            }

            /* Single stream with only two top-level match arms, but lots of sub-match arms */
            select! {
                value = s1.next().fuse() => {
                    match value {
                        Some(val) => match val {
                            val if val < 10 => match val {
                                val if val < 10 => match val {
                                    val if val < 10 => match val {
                                        val if val < 10 => match val {
                                            val if val < 10 => match val {
                                                val if val < 10 => match val {
                                                    val if val < 10 => match val {
                                                        _ => {},
                                                    },
                                                    _ => {},
                                                },
                                                _ => {},
                                            },
                                            _ => {},
                                        },
                                        _ => {},
                                    },
                                    _ => {},
                                },
                                _ => {},
                            },
                            _ => {}
                        },
                        None => {},
                    }
                }
            }
        }

    };
    task::block_on(future);
}

[Pauan]

This is really just an issue with any Rust macro which generates too much code. This happens because Rust has a very low recursion limit.

You can fix this by adding this at the top of your main.rs or lib.rs:

#![recursion_limit="1024"]

[rubdos]

It's quite annoying though, I had to put it at 4096 before rustc stopped complaining. At that point, I started thinking I was doing something wrong! (Maybe I was...)

[Pauan]

@rubdos 4096 sounds quite large! Could you post what your select! code looks like?

[rubdos]

You can have a look here. I've just tried with 1024, which didn't work, while 2048 still seems to work. We just refactored that select! call a bit, it used to be waaaay to long. It still is, but it's a bit better now.

[Pauan]

@rubdos Yeah, that's a lot of code. So it doesn't surprise me that it would run into the recursion limits.

[Matthias247]

What confuses me here is how it actually relates to recursion. The linked code is more or less a long list of statements. But those don't seem to be recursive (like with other macros inside)?

@rubdos The select looks very cool for a main state-machine. However be aware that every wakeup of the task will lead to polling all 9 sub-futures in the select loop. Probably won't matter if those are small, but is worthwhile to note.

[rubdos]

@rubdos Yeah, that's a lot of code. So it doesn't surprise me that it would run into the recursion limits.

Yes, figured that, but I agree with @Matthias247:

What confuses me here is how it actually relates to recursion. The linked code is more or less a long list of statements. But those don't seem to be recursive (like with other macros inside)?

i.e., I (and probably Matthias too) wonder why/where select! recurses.

@rubdos The select looks very cool for a main state-machine. However be aware that every wakeup of the task will lead to polling all 9 sub-futures in the select loop. Probably won't matter if those are small, but is worthwhile to note.

Actually, they're all Streams that I want to concurrently poll. I'm unsure how to only wake up and poll the correct one. They all transform a common mutable state.

[cramertj]

This is issue unfortunately unsolvable (as far as I know) due to the current way that the proc-macro-hack crate works. When expression position procedural macros are stabilized, we can move select! to use those and this issue will be automatically resolved. In the meantime, I'm going to close this issue.

[rubdos]

@cramertj: What about a note in the select! docs saying something along the lines of "Warning: select! relies on proc-macro-hack, and may require to set the compiler's recursion limit very high, e.g. #![recursion_limit="1024"]"?

[cramertj]

Sure, I wouldn't object to a PR adding docs for that!

[coriolinus]

@cramertj

When expression position procedural macros are stabilized, we can move select! to use those

It looks like that stabilization will happen in 1.45, which is likely to drop next week sometime. Is there an approximate ETA available for when select! will be reworked to avoid the proc-macro-hack?

[taiki-e]

@coriolinus We don't have an exact msrv policy so I'm not sure exactly how long we have to wait, but we probably have to wait for 3 to 6 months after it stabilized in stable channel (because this crate is widely used).

[taiki-e]

Alternatively, it can probably submit a patch to proc-macro-hack that allows you to use normal #[proc_macro] in 1.45 and later.