step-40: update IO section #13721
Merged
step-40: update IO section #13721
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Update the documentation on how we generate output using MPI I/O (we originally wrote one vtu per process). Remove the limit of 32 MPI ranks, because it is confusing according to my students. I think this was put in place to avoid generating 1000+ output files, which is no longer the case (we use 8 here). The output is a few MB in total, so I don't see a reason to limit things.
|
FYI @zjiaqi2018 @pengfej @singima |
bangerth
requested changes
May 11, 2022
examples/step-40/step-40.cc
Outdated
Comment on lines
545
to
558
| // bottleneck in parallelizing. In step-18, we have moved this step out of | ||
| // the actual computation but shifted it into a separate program that later | ||
| // combined the output from various processors into a single file. But this | ||
| // doesn't scale: if the number of processors is large, this may mean that | ||
| // the step of combining data on a single processor later becomes the | ||
| // longest running part of the program, or it may produce a file that's so | ||
| // large that it can't be visualized any more. We here follow a more | ||
| // sensible approach, namely creating individual files for each MPI process | ||
| // and leaving it to the visualization program to make sense of that. | ||
| // doesn't scale for several reasons: First, creating a single file per | ||
| // processor will overwhelm the filesystem with a large number of processors. | ||
| // Second, the step of combining all output files later can become the longest | ||
| // running part of the program, or it may produce a file that's so large that | ||
| // it can't be visualized easily any more. | ||
| // | ||
| // We here follow a more sophisticated approach that uses | ||
| // high-performance, parallel IO routines using MPI I/O to write to | ||
| // a small, fixed number of visualization files (here 8). We also | ||
| // generate a .pvtu record referencing these .vtu files, which can | ||
| // be opened directly in visualizatin tools like ParaView and VisIt. |
There was a problem hiding this comment.
Actually, the text in lines 545-547 is no longer true. In step-18 we now just create one file per process. Can you replcae the sentence "In step-18, we have moved this step out of the actual computation but shifted it into a separate program that later combined the output from various processors into a single file." by the following:
In those two programs, we simply generate one output file per process. That worked because the parallel::shared::Triangulation cannot be used with large numbers of MPI processes anyway.
Then also remove the "Second, ..." text since it's no longer what we do.
|
updated. |
bangerth
approved these changes
May 22, 2022
mkghadban
pushed a commit
to OpenFCST/dealii
that referenced
this pull request
Sep 8, 2022
step-40: update IO section
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update the documentation on how we generate output using MPI I/O (we
originally wrote one vtu per process).
Remove the limit of 32 MPI ranks, because it is confusing according to
my students. I think this was put in place to avoid generating 1000+
output files, which is no longer the case (we use 8 here). The output is
a few MB in total, so I don't see a reason to limit things.