Make The_Device_Arena non-managed (AMReX-Codes#2998)

The_Device_Arena used to be a separate Arena. We changed it to be an
alias of The_Arena to avoid memory fragmentation. However, the issue is
we don't have an Arena that can allocate non-managed memory unless
The_Arena is not managed. Because of performance concerns, we sometimes
want to allocate non-managed memory. Therefore, we make The_Device_Arena
an alias if and only if The_Arena is not managed.

Src/Base/AMReX_Arena.cpp

@@ -305,7 +305,7 @@ Arena::Initialize ()
     the_async_arena = new PArena(the_async_arena_release_threshold);
 
 #ifdef AMREX_USE_GPU
-    if (the_arena->isDevice() || the_arena->isManaged()) {
+    if (the_arena->isDevice()) {
         the_device_arena = the_arena;
     } else {
         the_device_arena = new CArena(0, ArenaInfo{}.SetDeviceMemory().SetReleaseThreshold

Src/Base/AMReX_GpuContainers.H

@@ -19,9 +19,8 @@ namespace Gpu {
 
     /**
      * \brief A PODVector that uses the standard memory Arena.
-     * Note that, on NVIDIA architectures, this Arena is actually
-     * managed.
-     *
+     * Note that the memory might or might not be managed depending
+     * on the amrex.the_arena_is_managed ParmParse parameter.
      */
     template <class T>
     using DeviceVector = PODVector<T, ArenaAllocator<T> >;