Hypre, Umpire and hipMemGetInfo

[PaulMullowney]

I'm not sure if this is the right place to ask this question. I'm using Hypre with Umpire enabled for AMD GPUs (rocm 5.1.0). In my application, I query the amount of free memory available with hipMemGetInfo after key steps (matrix creation, vector creation, preconditioner setup, ...). The results of these calls when not using Umpire are:

Before BuildMatrix : rank 0 of 8, free memory=63.98 GB, total memory=63.98 GB
Before hypre_CSRMatrixGpuRawSpMVAnalysis : rank 0 of 8, free memory=55.32 GB, total memory=63.98 GB
Before RHS IJVectorCreate : rank 0 of 8, free memory=54.90 GB, total memory=63.98 GB
After Preconditioner Setup : rank 0 of 8, free memory=43.68 GB, total memory=63.98 GB

This seems reasonable to me given the size of the problem. Once I build with Hypre with Umpire and a 4GB Device Memory Pool, I see:

Before BuildMatrix : rank 0 of 8, free memory=63.98 GB, total memory=63.98 GB
Before hypre_CSRMatrixGpuRawSpMVAnalysis : rank 0 of 8, free memory=.8517 GB, total memory=63.98 GB
Before RHS IJVectorCreate : rank 0 of 8, free memory=.8517 GB, total memory=63.98 GB
After Preconditioner Setup : rank 0 of 8, free memory=17.73 GB, total memory=63.98 GB

My question is, should I expect hipMemGetInfo to work properly when using device memory pools?
Thanks
-Paul

[PaulMullowney]

Anyone have any insight on this?

As a follow up, should I expect device memory pools larger than 4GB to work?

[davidbeckingsale]

Sorry for the delay. I think that it should work, but I'm not sure exactly how hypre (or you) are configuring and managing the pools. Can you share more details?