[BUG] Arena allocator doesn't handle per thread default streams and other streams allocating at the same time

[tgravescs]

Describe the bug
In Spark with the Spark Rapids accelerator using cudf 23.12 snapshot we have an application that is reading ORC files, doing some light processing and then writing ORC files. It consistently fails while doing the ORC write with:

terminate called after throwing an instance of 'rmm::logic_error'
  what():  RMM failure at:/home/jenkins/agent/workspace/jenkins-spark-rapids-jni_nightly-dev-594-cuda11/thirdparty/cudf/cpp/build/_deps/rmm-src/include/rmm/mr/device/arena_memory_resource.hpp:238: allocation not found

I believe the above issues stems from the change to orc writer to use the stream pool: rapidsai/cudf@cb06c20

Spark is running with per default stream turned on and now the orc writer has its own stream pool that is not per default stream.

Debugging the RMM code it appears that the following is happening:

1. a thread using a per thread default stream allocates memory and creates a thread Arena.
2. later someone else tries to allocate memory but it fails and causes a defrag. https://github.com/rapidsai/rmm/blob/branch-23.12/include/rmm/mr/device/arena_memory_resource.hpp#L160 , which calls clean() https://github.com/rapidsai/rmm/blob/branch-23.12/include/rmm/mr/device/arena_memory_resource.hpp#L177
3. The defrag sends all the superblocks_ from each of the arenas including the per thread default stream arena from step 1 back to the global arena: https://github.com/rapidsai/rmm/blob/branch-23.12/include/rmm/mr/device/detail/arena.hpp#L862
4. now one of the new streams from the orc writer pool allocates memory. This happens to move the superblock from the global arena into the stream arena.
5. The original thread using per thread default stream tries to deallocate the memory (https://github.com/rapidsai/rmm/blob/branch-23.12/include/rmm/mr/device/arena_memory_resource.hpp#L194) which isn't in the per thread default arena anymore so it falls into try the other arenas (https://github.com/rapidsai/rmm/blob/branch-23.12/include/rmm/mr/device/arena_memory_resource.hpp#L214). This code though check to see if use_per_thread_arena is on for that stream (https://github.com/rapidsai/rmm/blob/branch-23.12/include/rmm/mr/device/arena_memory_resource.hpp#L228), which it is and it only check the thread arenas to be deallocated. This fails and it falls through to try the global_arena (https://github.com/rapidsai/rmm/blob/branch-23.12/include/rmm/mr/device/arena_memory_resource.hpp#L238) which fails since the allocation is in the stream areans and it throws the "allocation not found"

I believe the fix for this is that since we are now mixing per thread default streams with normal streams it has to check all the arenas - threads and streams.

I have tested the above fix and it is no longer able to reproduce the issue.

Steps/Code to reproduce bug
I have no come up with a minimum repro outside Spark yet but I'm working on it.

Expected behavior
It doesn't crash.

Environment details (please complete the following information):

• Environment location: [Bare-metal, Docker, Cloud(specify cloud provider)]
• Method of RMM install: [conda, Docker, or from source]
  • If method of install is [Docker], provide docker pull & docker run commands used
• Please run and attach the output of the rmm/print_env.sh script to gather relevant environment details

Additional context
Add any other context about the problem here.