Problems in setting Multifab with ParallelFor #3970

solarmaterial · 2024-06-02T11:58:05Z

solarmaterial
Jun 2, 2024

Hey AMReX team,
I am trying to set data in the Multifab rhs based on the data from the Multifab phi_old. When the value of the phi(i,j,k,0) is around 0.001, I'd like to store the index number of the x-direction i0, and compute the value of the physical domain. Then another index number i1 is calculated. In the physical domain defined by these two index numbers, e.g. (x1,x0), I want to set rhs_array(i,j,k) = 1.

GpuArray<Real,AMREX_SPACEDIM> dx = geom.CellSizeArray();
GpuArray<Real,AMREX_SPACEDIM> prob_lo = geom.ProbLoArray();

rhs.setVal(0,0);
for ( MFIter mfi(rhs); mfi.isValid(); ++mfi )
 {
     const Box& bx = mfi.validbox();
     const Array4<Real>& rhs_array = rhs.array(mfi);
     const Array4<Real>& phi = phi_old.array(mfi);

     ParallelFor(bx, [=] AMREX_GPU_DEVICE(int i, int j, int k)
     {
         Real x0,x1;
         int i0, i1;      

         if ( phi(i,j,k,0) - 0.001 < 0. && phi(i+1,j,k,0) - 0.001 > 0.)
         {
             i0 = i;
             x0 = prob_lo[0] + (i0+Real(0.5)) * dx[0];
             x1 = x0 - 2.e-3;
             i1 = (x1 - prob_lo[0]) / dx[0] - Real(0.5);
         }

         if (i1< i  &&  i <= i0)
         {
          rhs_array(i,j,k) = 1.;
         }
                          
     });
 }

However, when then_cell is not equal to max_grid_size, such as when n_cell = 128, max_grid_size=32, the output of the code above is not what I expect. This might be the compute domain is cut to 4 parts, I guess? I wonder what I can do to fix this issue. I'm really looking forward to your reply!

Yours sincerely,
solar

Answered by WeiqunZhang

Jun 9, 2024

This was what I wrote " How many ghost cells does rhs have? From the various pieces you have shown, it probably needs 5. And after you fill rhs, you will need to call SumBounary so that the ghost cell values can be added to valid cells in other boxes. The special point might be near the box boundary such that i+1, or i+2, ... or i+5 is in the ghost cell region." More concisely, you need (1) ghost cells (2) SumBoundary call.

View full answer

WeiqunZhang · 2024-06-02T17:48:41Z

WeiqunZhang
Jun 2, 2024
Maintainer

If the first if test is false, i0 and i1 are uninitialized. That could be the issue.

(I assume rhs has already be initialized. Otherwise if the second if test is false, rhs in those cells is also uninitialized.)

13 replies

WeiqunZhang Jun 4, 2024
Maintainer

I've also uploaded a simple code to github.

But could you share a link so that we can better help you? How many ghost cells does rhs have? From the various pieces you have shown, it probably needs 5. And after you fill rhs, you will need to call SumBounary so that the ghost cell values can be added to valid cells in other boxes. The special point might be near the box boundary such that i+1, or i+2, ... or i+5 is in the ghost cell region.

solarmaterial Jun 4, 2024
Author

The link is here: https://github.com/solarmaterial/amrex
Thanks a lot!

WeiqunZhang Jun 4, 2024
Maintainer

If the first if test is false, i0 and i1 are uninitialized. That could be the issue.

My first comment still holds true. More specifically, the first if test is https://github.com/solarmaterial/amrex/blob/e11cb70c6af7b2d4f97dfc1c357b5eefaf18b87d/test/main.cpp#L86. If that is false, the code is equivalent to

int i0;
if (false) {
   i0 = i; // this line will not be run
}
if (i0 < ....) // i0 is uninitialized

solarmaterial Jun 7, 2024
Author

Thanks for all your replies! I'll definitely consider your suggestions. Thanks again for your kindness!

solarmaterial Jun 9, 2024
Author

Sorry to bother you again, but I still have some questions. I carefully checked the condition of the first if test according to your suggestion and used a new method to confirm whether the test was true. I have uploaded the updated code to GitHub.https://github.com/solarmaterial/amrex/tree/main/test

I set the values of phi_old to change only with x (i.e., i), and the output value is set to the integer value of the index:
phi_old_array(i,j,k,0) = i;
In this way, the 1D slice output of phi_old along the x-direction should be 0, 1, 2, 3, ... 31.

Now, I give the condition of the if statement:

if (7 < phi_old_array(i,j,k,0) && phi_old_array(i,j,k,0) < 9)
{
    rhs_array(i,j,k) = 9999; // i = 8
}

It is clear to see that the condition is met when phi_old_array(i,j,k,0) = 8, and at this time rhs_array(i,j,k) = 9999.
I confirm whether the condition is met by checking the 1D slice output of rhs along the x-direction. Since the value at the position with index 8 is 9999, the condition is successfully met.

My question still lies in the relationship between max_grid_size and n_cell. If I set n_cell = 32 and max_grid_size = 8, it is easy to understand that the computational domain is divided into 4 boxes: 0-7, 8-15, 16-23, 24-31.

At this time, the node we set earlier is exactly at index 8, which is the lower boundary of the second box 8-15. I give the additional code :

if (7 < phi_old_array(i,j,k,0) && phi_old_array(i,j,k,0) < 9)
{
    rhs_array(i,j,k) = 9999; // i = 8
    rhs_array(7,j,k) = 8888; // i-1 = 7
    rhs_array(i+1,j,k) = 10000; // i+1 = 9
    rhs_array(i-2,j,k) = 7777; // i-2 = 6
    rhs_array(i+2,j,k) = 11000; // i+2 = 10
}

Only the code for i greater than or equal to 8 is successfully executed, and the output result is:

0     0.015625     0
1     0.046875     0
...
7     0.234375     0
8     0.265625     9999
9     0.296875     10000
10    0.328125     11000
11    0.359375     0
...
31    0.984375     0

Since 9999 is present in the output, the test if (7 < phi_old_array(i,j,k,0) && phi_old_array(i,j,k,0) < 9) should have been successfully evaluated. Therefore, I guess the reason why the code for i less than 8 was not executed can only be that they are in different boxes.
(
If the index i is greater than 15, and it falls into the third or fourth box, the code remains unexecuted)

Do you have any suggestions for this issue? Thank you very much for your help!

WeiqunZhang · 2024-06-09T14:17:53Z

WeiqunZhang
Jun 9, 2024
Maintainer

#3970 (reply in thread)

…

On Sun, Jun 9, 2024, 5:49 AM solarmaterial ***@***.***> wrote: Sorry to bother you again, but I still have some questions. I carefully checked the condition of the first if test according to your suggestion and used a new method to confirm whether the test was true. I have uploaded the updated code to GitHub. https://github.com/solarmaterial/amrex/tree/main/test I set the values of phi_old to change only with x (i.e., i), and the output value is set to the integer value of the index: phi_old_array(i,j,k,0) = i; In this way, the 1D slice output of phi_old along the x-direction should be 0, 1, 2, 3, ... 31. Now, I give the condition of the if statement: if (7 < phi_old_array(i,j,k,0) && phi_old_array(i,j,k,0) < 9) { rhs_array(i,j,k) = 9999; // i = 8 } It is clear to see that the condition is met when phi_old_array(i,j,k,0) = 8, and at this time rhs_array(i,j,k) = 9999. I confirm whether the condition is met by checking the 1D slice output of rhs along the x-direction. Since the value at the position with index 8 is 9999, the condition is successfully met. My question still lies in the relationship between max_grid_size and n_cell. If I set n_cell = 32 and max_grid_size = 8, it is easy to understand that the computational domain is divided into 4 boxes: 0-7, 8-15, 16-23, 24-31. At this time, the node we set earlier is exactly at index 8, which is the lower boundary of the second box 8-15. I give the additional code : if (7 < phi_old_array(i,j,k,0) && phi_old_array(i,j,k,0) < 9) { rhs_array(i,j,k) = 9999; // i = 8 rhs_array(7,j,k) = 8888; // i-1 = 7 rhs_array(i+1,j,k) = 10000; // i+1 = 9 rhs_array(i-2,j,k) = 7777; // i-2 = 6 rhs_array(i+2,j,k) = 11000; // i+2 = 10 } Only the code for i greater than or equal to 8 is successfully executed, and the output result is: 0 0.015625 0 1 0.046875 0 ... 7 0.234375 0 8 0.265625 9999 9 0.296875 10000 10 0.328125 11000 11 0.359375 0 ... 31 0.984375 0 Since 9999 is present in the output, the test if (7 < phi_old_array(i,j,k,0) && phi_old_array(i,j,k,0) < 9) should have been successfully evaluated. Therefore, I guess the reason why the code for i less than 8 was not executed can only be that they are in different boxes. ( If the index i is greater than 15, and it falls into the third or fourth box, the code remains unexecuted) Do you have any suggestions for this issue? Thank you very much for your help! — Reply to this email directly, view it on GitHub <#3970 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB37TYKZGOOGMODPY5WKVY3ZGRFNBAVCNFSM6AAAAABIU742KCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TOMJYHE4TQ> . You are receiving this because you commented.Message ID: ***@***.***>

1 reply

solarmaterial Jun 9, 2024
Author

Since 9999 is present in the output, the test if (7 < phi_old_array(i,j,k,0) && phi_old_array(i,j,k,0) < 9) should have been successfully evaluated.

The first if test should have been true. Did I miss something?

WeiqunZhang · 2024-06-09T14:55:38Z

WeiqunZhang
Jun 9, 2024
Maintainer

Again please read #3970 (reply in thread) You asked for a suggestion. My suggestion was there in that link.

…

On Sun, Jun 9, 2024, 7:31 AM solarmaterial ***@***.***> wrote: Since 9999 is present in the output, the test if (7 < phi_old_array(i,j,k,0) && phi_old_array(i,j,k,0) < 9) should have been successfully evaluated. The first if test should have been true. Did I miss something? — Reply to this email directly, view it on GitHub <#3970 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB37TYJQWXD4AZOLBEP7BN3ZGRRNRAVCNFSM6AAAAABIU742KCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TOMJZGQ2TM> . You are receiving this because you commented.Message ID: ***@***.***>

0 replies

WeiqunZhang · 2024-06-09T14:58:27Z

WeiqunZhang
Jun 9, 2024
Maintainer

This was what I wrote " How many ghost cells does rhs have? From the various pieces you have shown, it probably needs 5. And after you fill rhs, you will need to call SumBounary so that the ghost cell values can be added to valid cells in other boxes. The special point might be near the box boundary such that i+1, or i+2, ... or i+5 is in the ghost cell region." More concisely, you need (1) ghost cells (2) SumBoundary call.

…

On Sun, Jun 9, 2024, 7:55 AM Weiqun Zhang ***@***.***> wrote: Again please read #3970 (reply in thread) You asked for a suggestion. My suggestion was there in that link. On Sun, Jun 9, 2024, 7:31 AM solarmaterial ***@***.***> wrote: > Since 9999 is present in the output, the test if (7 < > phi_old_array(i,j,k,0) && phi_old_array(i,j,k,0) < 9) should have been > successfully evaluated. > > The first if test should have been true. Did I miss something? > > — > Reply to this email directly, view it on GitHub > <#3970 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AB37TYJQWXD4AZOLBEP7BN3ZGRRNRAVCNFSM6AAAAABIU742KCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TOMJZGQ2TM> > . > You are receiving this because you commented.Message ID: > ***@***.***> >

1 reply

solarmaterial Jun 12, 2024
Author

I carefully went through your reviews again and, yes, the issue was with the ghost cells. After I set enough ghost cells to the Multifab and called SumBoundary, the prints of the rhs was as I expected. Thank you for your patience! I truly appreciate that!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problems in setting Multifab with ParallelFor #3970

{{title}}

Replies: 4 comments 15 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Problems in setting Multifab with ParallelFor #3970

solarmaterial Jun 2, 2024

Replies: 4 comments · 15 replies

WeiqunZhang Jun 2, 2024 Maintainer

WeiqunZhang Jun 4, 2024 Maintainer

solarmaterial Jun 4, 2024 Author

WeiqunZhang Jun 4, 2024 Maintainer

solarmaterial Jun 7, 2024 Author

solarmaterial Jun 9, 2024 Author

WeiqunZhang Jun 9, 2024 Maintainer

solarmaterial Jun 9, 2024 Author

WeiqunZhang Jun 9, 2024 Maintainer

WeiqunZhang Jun 9, 2024 Maintainer

solarmaterial Jun 12, 2024 Author

solarmaterial
Jun 2, 2024

Replies: 4 comments 15 replies

WeiqunZhang
Jun 2, 2024
Maintainer

WeiqunZhang Jun 4, 2024
Maintainer

solarmaterial Jun 4, 2024
Author

WeiqunZhang Jun 4, 2024
Maintainer

solarmaterial Jun 7, 2024
Author

solarmaterial Jun 9, 2024
Author

WeiqunZhang
Jun 9, 2024
Maintainer

solarmaterial Jun 9, 2024
Author

WeiqunZhang
Jun 9, 2024
Maintainer

WeiqunZhang
Jun 9, 2024
Maintainer

solarmaterial Jun 12, 2024
Author