sharing one regex across many threads can lead to big slowdowns due to mutex contention

[BurntSushi]

To reproduce, create a Cargo.toml with this:

[package]
name = "regex-contention-repro-work"
version = "0.1.0"
edition = "2021"

[[bin]]
name = "repro"
path = "main.rs"

[dependencies]
anyhow = "1.0.66"
regex = "1.7.0"

And in the same directory, create a main.rs containing:

use std::{
    io::Write,
    sync::Arc,
    time::{Duration, Instant},
};

use regex::{Regex, RegexBuilder};

const ITERS: usize = 100_000;
const PATTERN: &str = "";
const HAYSTACK: &str = "ZQZQZQZQ";

#[derive(Debug)]
struct Benchmark {
    re: Regex,
    threads: u32,
}

impl Benchmark {
    fn cloned(&self) -> anyhow::Result<Duration> {
        let start = Instant::now();
        let mut handles = vec![];
        for _ in 0..self.threads {
            // When we clone the regex like this, it does NOT make a complete
            // copy of all of its internal state, but it does create an entirely
            // fresh pool from which to get mutable scratch space for each
            // search. Basically, a 'Regex' internally looks like this:
            //
            //   struct Regex {
            //     // Among other things, this contains the literal
            //     // prefilters and the Thompson VM bytecode
            //     // instructions.
            //     read_only: Arc<ReadOnly>,
            //     // Contains space used by the regex matcher
            //     // during search time. e.g., The DFA transition
            //     // table for the lazy DFA or the set of active
            //     // threads for the Thompson NFA simulation.
            //     pool: Pool<ScratchSpace>,
            //   }
            //
            // That is, a regex already internally uses reference counting,
            // so cloning it does not create an entirely separate copy of the
            // data. It's effectively free. However, cloning it does create
            // an entirely fresh 'Pool'. It specifically does not reuse pools
            // across cloned regexes, and it does this specifically so that
            // callers have a path that permits them to opt out of contention
            // on the pool.
            //
            // Namely, when a fresh pool is created, it activates a special
            // optimization for the first thread that accesses the pool. For
            // that thread gets access to a special value ONLY accessible to
            // that thread, where as all other threads accessing the pool get
            // sent through the "slow" path via a mutex. When a lot of threads
            // share the same regex **with the same pool**, this mutex comes
            // under very heavy contention.
            //
            // It is worth pointing out that the mutex is NOT held for the
            // duration of the search. Effectively what happens is:
            //
            //   is "first" thread optimization active?
            //   NO: mutex lock
            //       pop pointer out of the pool
            //       mutex unlock
            //   do a search
            //   is "first" thread optimization active?
            //   NO: mutex lock
            //       push pointer back into pool
            //       mutex unlock
            //
            // So in the case where "do a search" is extremely fast, i.e., when
            // the haystack is tiny, as in this case, the mutex contention ends
            // up dominating the runtime. As the number of threads increases,
            // the contention gets worse and worse and thus runtime blows up.
            //
            // But, all of that contention can be avoided by giving each thread
            // a fresh regex and thus each one gets its own pool and each
            // thread gets the "first" thread optimization applied. So the
            // internal access for the mutable scratch space now looks like
            // this:
            //
            //   is "first" thread optimization active?
            //   YES: return pointer to special mutable scratch space
            //   do a search
            //   is "first" thread optimization active?
            //   YES: do nothing
            //
            // So how to fix this? Well, it's kind of hard. The regex crate
            // used to use the 'thread_local' crate that optimized this
            // particular access pattern and essentially kept a hash table
            // keyed on thread ID. But this led to other issues. Specifically,
            // its memory usage scaled with the number of active threads using
            // a regex, where as the current approach scales with the number of
            // active threads *simultaneously* using a regex.
            //
            // I am not an expert on concurrent data structures though, so
            // there is likely a better approach. But the idea here is indeed
            // to make it possible to opt out of contention by being able to
            // clone the regex. Once you do that, there are **zero** competing
            // resources between the threads.
            //
            // Why not just do this in all cases? Well, I guess I would if I
            // could, but I don't know how. The reason why explicit cloning
            // permits one to opt out is that each thread is handed its own
            // copy of the regex and its own pool, and that is specifically
            // controlled by the caller. I'm not sure how to do that from
            // within the regex library itself, since it isn't really aware of
            // threads per se.
            let re = self.re.clone();
            handles.push(std::thread::spawn(move || {
                let mut matched = 0;
                for _ in 0..ITERS {
                    if re.is_match(HAYSTACK) {
                        matched += 1;
                    }
                }
                matched
            }));
        }
        let mut matched = 0;
        for h in handles {
            matched += h.join().unwrap();
        }
        assert!(matched > 0);
        Ok(Instant::now().duration_since(start))
    }

    fn shared(&self) -> anyhow::Result<Duration> {
        let start = Instant::now();
        let mut handles = vec![];
        // We clone the regex into an Arc but then share it across all threads.
        // Each thread in turn competes with the single regex's shared memory
        // pool for mutable scratch space to use during a search. This is what
        // ultimately caused this 'shared' benchmark to be much slower than the
        // 'cloned' benchmark when run with many threads. Indeed, profiling it
        // reveals that most of the time is spent in the regex internal 'Pool'
        // type's 'get' and 'get_slow' methods.
        let re = Arc::new(self.re.clone());
        for _ in 0..self.threads {
            let re = Arc::clone(&re);
            handles.push(std::thread::spawn(move || {
                let mut matched = 0;
                for _ in 0..ITERS {
                    if re.is_match(HAYSTACK) {
                        matched += 1;
                    }
                }
                matched
            }));
        }
        let mut matched = 0;
        for h in handles {
            matched += h.join().unwrap();
        }
        assert!(matched > 0);
        Ok(Instant::now().duration_since(start))
    }
}

fn main() -> anyhow::Result<()> {
    let threads: u32 = std::env::var("REGEX_BENCH_THREADS")?.parse()?;
    let re = RegexBuilder::new(PATTERN)
        .unicode(false)
        .dfa_size_limit(50 * (1 << 20))
        .build()?;
    let benchmark = Benchmark { re, threads };
    let which = std::env::var("REGEX_BENCH_WHICH")?;
    let duration = match &*which {
        "cloned" => benchmark.cloned(),
        "shared" => benchmark.shared(),
        unknown => anyhow::bail!("unrecognized REGEX_BENCH_WHICH={}", unknown),
    };
    writeln!(std::io::stdout(), "{:?}", duration)?;
    Ok(())
}

Now build and run the benchmark:

$ cargo build --release 
$ hyperfine "REGEX_BENCH_WHICH=cloned REGEX_BENCH_THREADS=16 ./target/release/repro" "REGEX_BENCH_WHICH=shared REGEX_BENCH_THREADS=16 ./target/release/repro"
Benchmark 1: REGEX_BENCH_WHICH=cloned REGEX_BENCH_THREADS=16 ./target/release/repro
  Time (mean ± σ):       6.5 ms ±   1.2 ms    [User: 55.1 ms, System: 3.8 ms]
  Range (min … max):     0.1 ms …  10.5 ms    254 runs
 
  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
 
Benchmark 2: REGEX_BENCH_WHICH=shared REGEX_BENCH_THREADS=16 ./target/release/repro
  Time (mean ± σ):     530.5 ms ±  12.6 ms    [User: 1886.3 ms, System: 4994.7 ms]
  Range (min … max):   514.2 ms … 552.4 ms    10 runs
 
Summary
  'REGEX_BENCH_WHICH=cloned REGEX_BENCH_THREADS=16 ./target/release/repro' ran
   81.66 ± 15.52 times faster than 'REGEX_BENCH_WHICH=shared REGEX_BENCH_THREADS=16 ./target/release/repro'

As noted in the comments in the code above, the only difference between these two benchmarks is that cloned creates a fresh Regex for each thread where as shared uses the same Regex across multiple threads. This leads to contention on a single Mutex in the latter case and overall very poor performance. (See the code comments above for more info.)

We can confirm this by looking at a profile. Here is a screenshot from perf for the cloned benchmark:

And now for the shared benchmark:

As we can see, in the shared benchmark, virtually all of the time is being spent locking and unlocking the mutex.

[BurntSushi]

Note: A prerequisite to understanding this comment is probably to read the comments in the code snippet above.

I am unsure of how to fix this or whether it's even possible to. The regex crate used to use thread_local for this before going back to a simpler mutex scheme, but the way thread_local works led to big memory "leaks" in certain environments. Namely, thread_local's memory usage, AIUI, scales with the number regexes used across multiple threads while the regex crate's current implementation only scales with the number of regexes used simultaneously across multiple threads. So in the former case, if you have A LOT of threads running and using regexes, even if they aren't all being used simultaneously, then you wind up with exorbitant memory usage. But in the latter case, only the number of threads executing simultaneous regex searches matters for how much memory is used.

My guess is that in order to fix this, we will need a more sophisticated concurrent data structure for providing mutable scratch space while also employing some kind of compacting strategy.

Alternatively, we could mark this as wontfix and simply ask folks to clone their regexes before sending them to separate threads. A Regex already uses reference counting internally, so this won't copy all of the read-only data inside a Regex. But it does effectively put you back into the same memory usage profile as thread_local: the amount of mutable scratch space you'll end up using is proportional to the total number of threads using a regex.

[osiewicz]

Is exposing lower-level API that allows one to pass in scratch space an option at all?
My use case is that I have a regex that might be called from multiple threads, though only one thread at a time should be evaluating a given regex. I could store a scratch space externally and then pass that in, not having to go through an internal synchronization in the process.

[Alphare]

In case anyone has the same issue I did of using regex in a very parallel context with rayon (which AFAIK has no way of letting you control the initialization of a thread, which makes sense given its magic threadpool), here's the simple workaround that removes the contention that regex itself is responsible for:

struct TLSRegex {
    base: regex::bytes::Regex,
    local: thread_local::ThreadLocal<regex::bytes::Regex>,
}

impl TLSRegex {
    pub fn is_match(&self, path: &[u8]) -> bool {
        self.local.get_or(|| self.base.clone()).is_match(path)
    }
}

Basically: clone the Regex for every thread lazily on first access. As detailed above, this reuses all of the "read-only" compilation of the regex, only creating new scratch space for the DFA for each thread, so the overhead is actually low (negligible depending on your use-case? I haven't seen it show up in profiling).

This is using the thread_local crate mentioned above, which comes with the warnings laid out by @BurntSushi above about memory usage, by design.

[BurntSushi]

Is exposing lower-level API that allows one to pass in scratch space an option at all?

It's technically an option, but not one I'd ever sign off on personally. The regex API is essentially duplicated for &str and &[u8]. Duplicating each of those again just to accept scratch space to work around a problem that one can already work around is not worth it.

But, yes, I do plan on exposing such an API! Just not in regex, but rather, in regex-automata. You can see what it might look like here: https://burntsushi.net/stuff/tmp-do-not-link-me/regex-automata/regex_automata/meta/struct.Regex.html#method.try_find (Excuse the lack of docs. I'm actively working on that component. You'll find more docs on the lower level regex engines. For example: https://burntsushi.net/stuff/tmp-do-not-link-me/regex-automata/regex_automata/hybrid/regex/struct.Regex.html#method.find

I'm also planning on exposing the underlying Pool type in regex-automata too: https://burntsushi.net/stuff/tmp-do-not-link-me/regex-automata/regex_automata/util/pool/struct.Pool.html

Although you will have to do some kind of synchronization if you're calling a regex from multiple threads and watch to share the same mutable scratch space across multiple threads. And if you're not running regexes simultaneously then there won't be any contention and you won't be bitten by this issue in the first place.

The regex-automata plans are tracked by #656.

only creating new scratch space for the DFA for each thread, so the overhead is actually low (negligible depending on your use-case? I haven't seen it show up in profiling).

Right. The thread_local crate will give you speed, as long as you can deal with its memory profile. (AFAIK, its memory profile is only an issue when you have lots of threads. If you keep your thread pool limited to some reasonable number then I don't think you can run into any problems. Problems only tended to appear in managed environments, e.g., using the regex crate from a Go program.)

[BurntSushi]

Also note that the design space is larger than what I've implemented. RE2, for example, shares the same mutable scratch space across multiple threads simultaneous. But that means you now have atomics and mutexes and overhead inside the lazy DFA search code itself. It's not a path I want to go down personally.

[WGH-]

I believe 1.4.3 is the last version that doesn't perform poorly in multithreaded scenarios?

[BurntSushi]

I wouldn't say that. The current version of regex doesn't "perform poorly in multi-threaded scenarios." That's far too broad of a statement.

regex 1.4.3 is the last release to have an optional dependency on thread_local. In some cases it can perform better in highly contended scenarios where you're searching short haystacks. But it can also result in exorbitant memory usage in some cases, which was why that strategy was removed.

You're also ignoring the fact that you can clone the Regex when sending it to another thread. This won't use any extra memory and it eliminates the contention problem.

[WGH-]

The current version of regex doesn't "perform poorly in multi-threaded scenarios." That's far too broad of a statement.

Right. I apologize for that.

In some cases it can perform better in highly contended scenarios where you're searching short haystacks. But it can also result in exorbitant memory usage in some cases, which was why that strategy was removed.

My application is exactly that: throwing lots of threads (rayon-style) at matching regexes.

And I also agree that unavoidable memory leaks (in some cases) is far worse than work-aroundable performance degradation.

Still, I think it's important to have a point of reference that one may compare to. I made a decision to downgrade and pin regex version for the time being (I initially noticed when I cargo updated everything), and maybe other people want to, as well.

[BurntSushi]

Pinning to old and unsupported version of regex doesn't seem like a good idea and certainly not one that should be promoted or even suggested.

[koute]

Alternatively, we could mark this as wontfix and simply ask folks to clone their regexes before sending them to separate threads.

Maybe it could be nice to have a feature on regex which would control whether Regexes are Sync? Otherwise this issue is somewhat of a trap and it's only going to get worse as more people get hugely mulicore machines. On my machine (32 cores/64 threads) the performance degradation is so bad that I'd probably go as far as calling it broken. It'd be nice to be able to statically make sure that the scratch space is never implicitly shared.

[BurntSushi]

Maybe it could be nice to have a feature on regex which would control whether Regexes are Sync?

I'm pretty sure that this can never happen. It can be accomplished in two ways. The first is to add a new non-default feature that, when present, causes Regex to not implement Sync. Imagine a case where a library foo enables this feature. Then anyone depending on it and using regex elsewhere will also get that feature enabled. And that can in turn introduce a build failure that will be impossible to work-around without fixing foo.

The second way is to add a new feature sync that is enabled by default, that when turned off will result in Regex not implementing Sync. The problem there is that regex = { version = "1.8", default-features = false, features = ["std"] } is a supported configuration today. But with this new feature, folks using that configuration could very well end up getting a build failure because they don't have sync in the feature list.

So this idea is a dead end as far as I can see. Even if one did find a way to do this with crate features, I do not like this path at all. I would need some pretty compelling testimony to go down this path, including exhausting alternatives.

Otherwise this issue is somewhat of a trap and it's only going to get worse as more people get hugely mulicore machines.

The issue, AIUI, is not "uses a single regex from many threads simultaneously." My understanding is still that the unit of work has to be small, or else there shouldn't be an opportunity for so much contention.

broken

Can we please not do this hyperbole here? This word is waaaaaay over-used. Its use isn't going to have a positive effect here.

It'd be nice to be able to statically make sure that the scratch space is never implicitly shared.

Once #656 is done and regex-automata 0.3 is out, you should be able to do this with a meta::Regex from regex-automata. See the docs here: https://burntsushi.net/stuff/tmp-do-not-link-me/regex/regex_automata/meta/struct.Regex.html#synchronization-and-cloning

I'd also like to point out that nobody has tried to actually fix the performance problem. Literally zero attempts. I would love for an expert in concurrent data structures to sink their teeth into this problem. Once #656 is done, I can try to advertise this need more broadly. But I don't have the bandwidth to mentor it right now before #656.

[koute]

Can we please not do this hyperbole here? This word is waaaaaay over-used. Its use isn't going to have a positive effect here.

Sorry, it wasn't my intention to use hyperbole, it's just, with all due respect, the performance difference between "finishes in 10 minutes" and "not finishes in 24 hours" (which I have personally encountered due to this issue; only once I left my PC running overnight and discovered in the morning that my program didn't finish I attached a profiler, analyzed it and ended up here in this issue) does qualify as pretty much broken in my opinion. If a program is so slow that it can't give me an answer in a reasonable time then it's just as if it couldn't give it to me at all, i.e. just as if it was broken. Especially since this is a fairly subtle trap that's relatively easy to trigger (especially with e.g. rayon-like concurrency) and relatively tricky to diagnose (it's not immediately obvious that any locks are involved, and you need to use a profiler to see what's going on). So this was not an attempt at hyperbole; from my perspective it literally broke my program because it couldn't give me an answer in a reasonable time. (: Sorry if it came out that way.

[BurntSushi]

@koute It would just be better to focus on what's actually happening instead of just saying "broken." Pretty much everyone has a reason why they use the word "broken," but the word "broken" doesn't actually help anyone understand anything.

I would personally love to get a repro of your particular issue. "finishing in minutes" versus "finishing in days" is indeed a very big difference and I would love to have a reproduction for that so that myself (or others) have a way of getting a real use case on in their hands to optimize for.

I agree it's a trap that one can fall into, but again and I've said now, my understanding is that "just using rayon" isn't enough to provoke this issue. But maybe my mental model needs updating. Without reproductions, that's hard.

[koute]

It would just be better to focus on what's actually happening instead of just saying "broken." Pretty much everyone has a reason why they use the word "broken," but the word "broken" doesn't actually help anyone understand anything.

Fair enough. Apologies. I should have immediately explained myself.

I agree it's a trap that one can fall into, but again and I've said now, my understanding is that "just using rayon" isn't enough to provoke this issue. But maybe my mental model needs updating. Without reproductions, that's hard.

AFAIK from what you've described your mental model of the issue is mostly correct, or at least it matches what I've seen. Basically, run on hugely multicore machine, saturate every hardware thread (I have 64 hardware threads) and do a lot of (small) regex matches.

I can't really share the code of my exact case, but I actually might have a reasonable real world test case for you, as it just happens that recently during the weekend I encountered this issue in one of the crates in the wild and put up a PR optimizing it.

1. Set up a reproduction and download some test data:

$ cargo new --bin repro
$ cd repro
$ cargo add lingua --git https://github.com/koute/lingua-rs.git --rev 92f8d5d1e078cfafcbc60d52b6f6821a65ded422
$ cargo add rayon
$ curl "https://www.gutenberg.org/cache/epub/345/pg345.txt" -o src/dracula.txt

2. Paste this in src/main.rs:

use rayon::prelude::*;

fn main() {
    let data = include_str!("dracula.txt");
    let det = lingua::LanguageDetectorBuilder::from_all_languages().build();
    let xs: Vec<_> = std::iter::repeat(data).take(100000).collect();
    xs.into_par_iter().for_each(|s| {
        let len = s.chars().map(|ch| ch.len_utf8()).take(2048).sum::<usize>();
        std::hint::black_box(det.detect_language_of(&s[..len]));
    });
}

3. Run it:

$ cargo build --release && time cargo run --release

real	2m0.792s
user	23m49.315s
sys	96m4.691s

4. Now replace the dependency with one which doesn't use regex for the hot path (so there's no lock contention happening):

$ cargo remove lingua
$ cargo add lingua --git https://github.com/koute/lingua-rs.git --rev 304cb5a113ddbb968fab2ef45e41b686e42e5aa8

5. Rerun it:

$ cargo build --release && time cargo run --release

real	0m5.737s
user	5m23.120s
sys	0m0.247s

And just for reference, here are the numbers when running on a single thread (basically just replace into_par_iter with into_iter):

# Before my optimizations (uses regex):
real	3m39.142s
user	3m39.041s
sys	0m0.100s

# After my optimizations (doesn't use regex):
real	3m35.342s
user	3m35.204s
sys	0m0.043s

So when using regex jumping from 1 thread to 64 threads the program only got 1m39s faster (~54% of what it was) but wasted over an hour and half of CPU time. When not using regex jumping from 1 thread to 64 threads the program got 3m30s faster (~2% of what it was) and wasted less than two minutes of CPU time. When running single-threaded we can see that my changes didn't affect the non-contended case too much.

Of course these are the results on my machine (Threadripper 3970x, 256GB of 2667 MT/s RAM); your mileage may vary.

And here's the commit which got rid of regex usage so that you can see for what exactly that crate is using regex: koute/lingua-rs@304cb5a

You could argue that maybe it's not an appropriate use of regex, but nevertheless this is what I've encountered in the wild.