Upgrade write_vtu_in_parallel based on mpi large IO update #13673

pengfej · 2022-05-05T00:17:26Z

Replace the MPI I/O routines for DataOut::write_vtu_in_parallel:

Use the large count facility to support >2GB+ individual writes
Use write_at_all instead of write_shared (this is up to 10x faster in testing on Frontera)

tjhei · 2022-05-05T00:40:21Z

This is WIP and depends on #13611

pengfej · 2022-05-07T23:24:03Z

I will remove all the extra comments and space after all so don't worry!

source/base/data_out_base.cc

bangerth · 2022-05-25T23:28:49Z

source/base/data_out_base.cc

+      ierr = MPI_File_write_at(fh,
+                               footer_offset,
+                               ss.str().c_str(),
+                               footer_size,
+                               MPI_CHAR,
+                               MPI_STATUS_IGNORE);


Is this the right function call, or don't you need one from the big_mpi library?

To the best of my knowledge, the header and footer are very small in terms of size, so using write_at and write_at_c (The one in big_mpi library) will not make a difference :)

I agree with Pengfei.

Could we do it just out of consistency and so that people like me don't have to think about the question in the future :-)

source/base/data_out_base.cc

tjhei

Looks good. Can you please squash your commits?

tjhei · 2022-05-26T16:21:59Z

Let me test this some more before merging...

tjhei · 2022-05-26T16:30:48Z

@pengfej We forgot to test one important case: if you run the code with n_ranks == 1, then rank 0 will try to send to itself, which causes a hang. You can see this with tests/mpi/parallel_vtu_01.mpirun=1.debug, which does not complete.

tjhei · 2022-05-26T16:36:12Z

My suggestion would be to create a new block before if (myrank == 0) that does something like:

if (myrank ==0 || myrank == n_ranks -1)
{
ierr = MPI_Sendrecv_replace(&footer_offset, 1, MPI_UINT64_T, n_ranks - 1, 0, 0, 0, comm, MPI_STATUS_IGNORE);
}

You will need to delete the other MPI_Send and MPI_Recv but keep the assignment of footer_offset on rank 0 (and move the declaration of this variable up and outside the if).

MPI_Sendrecv_replace is a combination of MPI_Send and MPI_Recv and replaces the contents in the buffer given, which is exactly what we need.

pengfej · 2022-05-26T16:39:01Z

@tjhei I see, thanks a lot, let me give it a try.

pengfej · 2022-05-26T17:53:40Z

@tjhei rank=1 tests passed, but rank=3 test freezes for a very long time, I'm not very sure if it's a code error or my machine is not working properly.

tjhei · 2022-05-26T18:08:32Z

source/base/data_out_base.cc

+    // Sending the offset for writing the footer to rank 0.
+    if (myrank == n_ranks - 1)
+      {
+        const std::uint64_t footer_offset = size_on_proc + offset;
+        AssertThrowMPI(ierr);
+      }


Suggested change

// Sending the offset for writing the footer to rank 0.

if (myrank == n_ranks - 1)

{

const std::uint64_t footer_offset = size_on_proc + offset;

AssertThrowMPI(ierr);

}

if (myrank == n_ranks - 1)

footer_offset = size_on_proc + offset;

You never set the footer_offset correctly (you ended up creating a local variable here and setting it).

I see. Let me changed that and test it again, thanks for pointing out!

The test mpirun=3 still freezes, I'm wondering if there's a way to know where exactly it freezes.

There is, of course. I will show you next time.

sounds good! I just read about the sendrecv_replace function and it seems like I might misordered the sender and the receiver, I'm testing on that fix now.

tjhei · 2022-05-26T18:08:52Z

source/base/data_out_base.cc

+                                  0,
+                                  comm,
+                                  MPI_STATUS_IGNORE);
+    }


add the AssertThrow here

tjhei · 2022-05-26T19:11:49Z

Can you switch source and dest of MPI_SendRecv_replace?

pengfej · 2022-05-26T19:16:06Z

Can you switch source and dest of MPI_SendRecv_replace?

Just did! The tests are still frozen unfortunately..

pengfej · 2022-05-26T19:44:05Z

The only part missing on the testing output is the last chunk, but I can't post that vtu code in this comment section for some reason.

tjhei

I have a simpler suggestion: Just write the footer from the last rank instead of rank 0. This way you don't need to do any communication and you can use the footer_offset direcly.

source/base/data_out_base.cc

tjhei · 2022-05-27T01:51:03Z

source/base/data_out_base.cc

+                                  MPI_STATUS_IGNORE);
+      AssertThrowMPI(ierr);
+    }
+
  // write footer
  if (myrank == 0)


Just write the footer on rank n_ranks-1 instead of 0.

That fixed version passed the test on my machine so I pushed it! Thank you so much!!

Updating author name fixing work update indent Remove the broadcast and fix some comment and fix author name updates updates! last fix fixing indent fixing unmatched int type fixing indent trying to fix the broadcast error Trying to fix broadcast error II adding _c to write_at functions fixing rank issue fixing footer_offset last fix on rank indent removing some unused line

tjhei

I can confirm that things work correctly now.

tjhei · 2022-05-29T19:22:23Z

This is ready to go from our side. Any final comments?

bangerth

Looks good. It tests alright, so I'll merge, but if you wouldn't mind, please make the comments I have about comments into a separate PR.

bangerth · 2022-05-30T03:38:09Z

source/base/data_out_base.cc

-      // shared file pointer to the location after header;
-      ierr = MPI_File_write_shared(
-        fh, ss.str().c_str(), header_size, MPI_CHAR, MPI_STATUS_IGNORE);
+      // Write the header on rank 0 at the starting of a file, i.e., offset 0.


Suggested change

// Write the header on rank 0 at the starting of a file, i.e., offset 0.

// Write the header on rank 0 at the start of a file, i.e., offset 0.

bangerth · 2022-05-30T03:38:42Z

source/base/data_out_base.cc

@@ -7741,21 +7746,47 @@ DataOutInterface<dim, spacedim>::write_vtu_in_parallel(
                                  vtk_flags,
                                  ss);

-    ierr = MPI_File_write_ordered(
-      fh, ss.str().c_str(), ss.str().size(), MPI_CHAR, MPI_STATUS_IGNORE);
+    // using prefix sum to find specific offset to write at.


Suggested change

// using prefix sum to find specific offset to write at.

// Use prefix sum to find specific offset to write at.

bangerth · 2022-05-30T03:39:59Z

source/base/data_out_base.cc

+        DataOutBase::write_vtu_footer(ss);
+        const unsigned int footer_size = ss.str().size();
+
+        // Writing Footer.


Suggested change

// Writing Footer.

// Writing footer:

tjhei · 2022-05-30T19:52:19Z

@pengfej Please create a follow-up with the fixes suggested by @bangerth , ok?

pengfej · 2022-05-30T20:10:17Z

@pengfej Please create a follow-up with the fixes suggested by @bangerth , ok?

Ohh sure! Sorry I completely missed it. I only saw the merged notification.

Upgrade write_vtu_in_parallel based on mpi large IO update

pengfej force-pushed the myowndealii branch from 22d3d98 to e318af3 Compare May 5, 2022 00:25

tjhei added the WIP ⚠️ label May 5, 2022

pengfej force-pushed the myowndealii branch from e5c4c26 to d077880 Compare May 7, 2022 23:18

pengfej force-pushed the myowndealii branch 2 times, most recently from d5cb7de to 0848d2c Compare May 8, 2022 21:40

tjhei added the ready to test label May 25, 2022

tamiko added this to the Release 9.4 milestone May 25, 2022

tamiko added the High Priority ⚠️ label May 25, 2022

pengfej force-pushed the myowndealii branch 4 times, most recently from 2ae3559 to e4b1579 Compare May 25, 2022 22:48

bangerth reviewed May 25, 2022

View reviewed changes

pengfej force-pushed the myowndealii branch 2 times, most recently from 2e6e346 to 28f1513 Compare May 26, 2022 00:48

tjhei reviewed May 26, 2022

View reviewed changes

source/base/data_out_base.cc Outdated Show resolved Hide resolved

tjhei removed the WIP ⚠️ label May 26, 2022

tjhei approved these changes May 26, 2022

View reviewed changes

pengfej force-pushed the myowndealii branch from a3d729f to ad13a8c Compare May 26, 2022 15:08

drwells approved these changes May 26, 2022

View reviewed changes

drwells added the Reviewed and ready to merge label May 26, 2022

tjhei removed the Reviewed and ready to merge label May 26, 2022

tjhei reviewed May 26, 2022

View reviewed changes

tjhei reviewed May 27, 2022

View reviewed changes

pengfej force-pushed the myowndealii branch from 61a6e71 to 44bdb8e Compare May 27, 2022 02:15

tjhei approved these changes May 27, 2022

View reviewed changes

bangerth approved these changes May 30, 2022

View reviewed changes

bangerth merged commit 802b3e0 into dealii:master May 30, 2022

pengfej mentioned this pull request May 30, 2022

Fixing grammar mistakes in comments #13863

Merged

tjhei mentioned this pull request Jun 1, 2022

Bug: writing large vtu files fails #12974

Closed

mkghadban pushed a commit to OpenFCST/dealii that referenced this pull request Sep 8, 2022

Merge pull request dealii#13673 from pengfej/myowndealii

3ba2b14

Upgrade write_vtu_in_parallel based on mpi large IO update

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade write_vtu_in_parallel based on mpi large IO update #13673

Upgrade write_vtu_in_parallel based on mpi large IO update #13673

pengfej commented May 5, 2022 •

edited by tjhei

tjhei commented May 5, 2022

pengfej commented May 7, 2022

bangerth May 25, 2022

pengfej May 25, 2022

tjhei May 26, 2022

bangerth May 26, 2022

pengfej May 26, 2022

tjhei left a comment

tjhei commented May 26, 2022

tjhei commented May 26, 2022

tjhei commented May 26, 2022 •

edited

pengfej commented May 26, 2022

pengfej commented May 26, 2022

tjhei May 26, 2022

tjhei May 26, 2022

pengfej May 26, 2022

pengfej May 26, 2022

tjhei May 26, 2022

pengfej May 26, 2022

tjhei May 26, 2022

tjhei commented May 26, 2022

pengfej commented May 26, 2022 •

edited

pengfej commented May 26, 2022 •

edited

tjhei left a comment

tjhei May 27, 2022

pengfej May 27, 2022

tjhei left a comment

tjhei commented May 29, 2022

bangerth left a comment

bangerth May 30, 2022

bangerth May 30, 2022

bangerth May 30, 2022

tjhei commented May 30, 2022

pengfej commented May 30, 2022

	// Write the header on rank 0 at the starting of a file, i.e., offset 0.
	// Write the header on rank 0 at the start of a file, i.e., offset 0.

	// using prefix sum to find specific offset to write at.
	// Use prefix sum to find specific offset to write at.

Upgrade write_vtu_in_parallel based on mpi large IO update #13673

Upgrade write_vtu_in_parallel based on mpi large IO update #13673

Conversation

pengfej commented May 5, 2022 • edited by tjhei

tjhei commented May 5, 2022

pengfej commented May 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tjhei left a comment

Choose a reason for hiding this comment

tjhei commented May 26, 2022

tjhei commented May 26, 2022

tjhei commented May 26, 2022 • edited

pengfej commented May 26, 2022

pengfej commented May 26, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tjhei commented May 26, 2022

pengfej commented May 26, 2022 • edited

pengfej commented May 26, 2022 • edited

tjhei left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tjhei left a comment

Choose a reason for hiding this comment

tjhei commented May 29, 2022

bangerth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tjhei commented May 30, 2022

pengfej commented May 30, 2022

pengfej commented May 5, 2022 •

edited by tjhei

tjhei commented May 26, 2022 •

edited

pengfej commented May 26, 2022 •

edited

pengfej commented May 26, 2022 •

edited