Update tutorials to use write_vtu_with_pvtu_record #8963

peterrum · 2019-10-25T23:54:57Z

some of the comments still have to be updated...

peterrum · 2019-10-26T11:17:09Z

examples/step-42/step-42.cc

-        data_out.write_pvtu_record(pvtu_master_output, filenames);
-
-        std::ofstream visit_master_output(
-          (output_dir + filename_base + ".visit"));


What do we want to do with this .visit file?

tjhei · 2019-10-26T13:43:47Z

you can drop step-50, see #8948

examples/step-18/step-18.cc

tjhei · 2019-10-26T13:44:45Z

examples/step-18/step-18.cc

-        DataOutBase::write_pvd_record(pvd_output, times_and_names);
-      }
+    // Let us open a file and write the data we have generated into it:
+    data_out.write_vtu_with_pvtu_record(


what happened to timestep_no?

tjhei · 2019-10-26T13:45:30Z

examples/step-32/step-32.cc

-        const std::string visit_master_filename =
-          ("solution-" + Utilities::int_to_string(out_index, 5) + ".visit");
-        std::ofstream visit_master(visit_master_filename);
-        DataOutBase::write_visit_record(visit_master, filenames);


what about the visit record, do we care? @bangerth are you using .pvtu or .visit with visit?

I'm using the .visit file. I think it is true that Visit can now read .pvd files, right?

tjhei · 2019-10-26T13:46:16Z

examples/step-40/step-40.cc

-      }
+    // The next step is to write this data to disk.
+    data_out.write_vtu_with_pvtu_record(
+      "./", "solution-", cycle, 2, mpi_communicator);


can we use MPI I/O here and explain it in a paragraph or so? Maybe use 8 or so?

peterrum · 2019-10-26T14:19:48Z

@tjhei Thanks for your comments! I have integrated them. The only question is what happens with the visit record. In step-50, you have eliminated it...

tjhei · 2019-10-26T14:26:05Z

I don't care about it, but I let Wolfgang comment if he needs them to use with visit. 😉

examples/step-18/step-18.cc

bangerth · 2019-10-26T14:47:33Z

On 10/26/19 8:26 AM, Timo Heister wrote: I don't care about it, but I let Wolfgang comment if he needs them to use with visit. 😉

Yes, please keep the .visit file. It's the equivalent of the .pvd file that Paraview uses.

peterrum · 2019-10-26T15:11:49Z

@bangerth I find it quite annoying to keep the .visit-stuff because for that we have to collect the filenames (which also happens within write_vtu_with_pvtu_record). The result is that the code will not be simpler as before...

Personally, I think there should be no write_vtu_with_pvtu_record() but there should be a DataOutFlags::Vtu (similarly as GridOutFlags::Vtu), which configures write_vtu() in regard of number of output files, .pvtu- and .visit-output (and other parameters). Or does anyone want to implement write_vtu_with_visit_record()?

tjhei · 2019-10-26T15:36:55Z

I find it quite annoying to keep the .visit-stuff because for that we have to collect t

okay, so the current .visit master file format lists all individual files at each timestep:

!TIME 0
!TIME 1
!TIME 2
!TIME 3
!NBLOCKS 4
solution/solution-00000.0000.vtu
solution/solution-00000.0001.vtu
solution/solution-00000.0002.vtu
solution/solution-00000.0003.vtu
solution/solution-00001.0000.vtu
solution/solution-00001.0001.vtu
solution/solution-00001.0002.vtu
solution/solution-00001.0003.vtu
solution/solution-00002.0000.vtu
solution/solution-00002.0001.vtu
solution/solution-00002.0002.vtu
solution/solution-00002.0003.vtu
solution/solution-00003.0000.vtu
solution/solution-00003.0001.vtu
solution/solution-00003.0002.vtu
solution/solution-00003.0003.vtu

I just played around with this, and this works for me in visit as well:

!TIME 0
!TIME 1
!TIME 2
!TIME 3
!NBLOCKS 1
solution/solution-00000.pvtu
solution/solution-00001.pvtu
solution/solution-00002.pvtu
solution/solution-00003.pvtu

This means we can make a new function to generate a visit master record (like the .pvd) that doesn't need the individual filenames but references the .pvtus!

edit: @peterrum I can make these changes to the master record function, if you want. This means you can delete everything related to .visit files here for now.

peterrum · 2019-10-26T16:43:45Z

examples/step-18/step-18.cc

@@ -1333,6 +1288,8 @@ namespace Step18
          std::pair<double, std::string>(present_time, pvtu_master_filename));
        std::ofstream pvd_output("solution.pvd");
        DataOutBase::write_pvd_record(pvd_output, times_and_names);
+        std::ofstream visit_output("solution.visit");
+        DataOutBase::write_visit_record(visit_output, times_and_names);


~~@tjhei Like this? In the other visit-cases this does not work. Here it is used as an equivalent for .pvd, there is is used as an equivalent for .pvtu.~~

peterrum · 2019-10-28T21:16:35Z

edit: @peterrum I can make these changes to the master record function, if you want. This means you can delete everything related to .visit files here for now.

@tjhei I have removed the .visit instances.

trailing dashes look ugly as the filename is "solution-_0001" etc.

tjhei · 2019-10-29T07:01:03Z

I pushed a couple of minor changes on top of yours.

To explain the situation, there are two different kind of records:

group pieces into a single time step (.pvtu can be read by visit and paraview and a .visit that can only be read by visit)
=> my vote would be to only write the .pvtu because:
1. fewer files
2. .pvtu is a standard, .visit is not
3. less code in tutorials
4. it is confusing to have two different kind of .visit files (see below)
There are master records that contain several time steps with times (.pvd and .visit, can only be read by paraview and visit, respectively).
=> it turns out non of the tutorials were writing this file in visit format. I added it for step-18 here (I can look at other examples as well)

@bangerth are you okay with this?

peterrum · 2019-10-29T18:36:40Z

@tjhei looks good! Thanks!

peterrum · 2019-10-31T22:57:45Z

Is there a way to test the tutorials?

tjhei · 2019-11-01T01:26:02Z

Is there a way to test the tutorials?

No, I am not aware of a way to do that easily.

bangerth · 2019-11-02T03:27:33Z

examples/step-18/step-18.cc

-    // we have generated into it:
-    std::ofstream output(filename);
-    data_out.write_vtu(output);
+    // Let us open a file and write the data we have generated into it:


Suggested change

// Let us open a file and write the data we have generated into it:

// Let us call a function that opens the necessary output files and writes the

// data we have generated into them. The function automatically constructs

// the file names from the given directory name (the first argument) and file

// name base (second argument). It augments the resulting string by pieces

// that result from the time step number and a "piece number" that corresponds

// to a part of the overall domain that can consist of one or more subdomains.

//

// The function also writes a record files (with suffix `.pvd`) for Paraview

// that describes how all of these output files combine into the data for

// this single time step:

bangerth · 2019-11-02T03:28:15Z

examples/step-18/step-18.cc

-    std::ofstream output(filename);
-    data_out.write_vtu(output);
+    // Let us open a file and write the data we have generated into it:
+    const auto pvtu_master_filename = data_out.write_vtu_with_pvtu_record(


Let's be explicit:

Suggested change

const auto pvtu_master_filename = data_out.write_vtu_with_pvtu_record(

const std::string pvtu_master_filename = data_out.write_vtu_with_pvtu_record(

bangerth · 2019-11-02T03:29:45Z

examples/step-18/step-18.cc

@@ -1333,6 +1288,13 @@ namespace Step18
          std::pair<double, std::string>(present_time, pvtu_master_filename));
        std::ofstream pvd_output("solution.pvd");
        DataOutBase::write_pvd_record(pvd_output, times_and_names);
+
+        std::ofstream visit_output("solution.visit");


Suggested change

std::ofstream visit_output("solution.visit");

// The final piece of this is to also write a master record for Visit, in the same way as the

// call above already did for Paraview:

std::ofstream visit_output("solution.visit");

bangerth · 2019-11-02T03:30:54Z

examples/step-32/step-32.cc

-        const std::string visit_master_filename =
-          ("solution-" + Utilities::int_to_string(out_index, 5) + ".visit");
-        std::ofstream visit_master(visit_master_filename);
-        DataOutBase::write_visit_record(visit_master, filenames);


I'm using the .visit file. I think it is true that Visit can now read .pvd files, right?

bangerth · 2019-11-02T03:31:55Z

examples/step-18/step-18.cc

+
+        std::ofstream visit_output("solution.visit");
+        static std::vector<std::pair<double, std::vector<std::string>>>
+          times_and_pieces;


The use of a static variable is really not very pleasant here because it means that you can't call the simulator twice in a row, or twice in parallel. I think it would be nicer to put this into a member variable instead.

But maybe we don't need this -- does Visit now read .pvd files?

bangerth · 2019-11-02T03:33:10Z

examples/step-32/step-32.cc

-        DataOutBase::write_visit_record(visit_master, filenames);
-      }
+    static int out_index = 0;
+    data_out.write_vtu_with_pvtu_record(


Same here with the static variable -- I recognize that that's preexisting, but if it were me, I'd say let's just make that a member variable with a more descriptive name.

can we do the static variables in a separate PR?

bangerth · 2019-11-02T03:34:06Z

examples/step-40/step-40.cc

+    // in parallel with the help of MPI-IO. Additionally a PVTU record is
+    // generated, which groups the written VTU files.
+    data_out.write_vtu_with_pvtu_record(
+      "./", "solution", cycle, 2, mpi_communicator, 8);


Is 8 a good number? We ran this program with up to 16k processors. I'd expect that with just 8 blocks we'd get into a rather bad bottleneck here!

just 8 blocks we'd get into a rather bad bottleneck here

It should not be a bottleneck. All 16k processes would write in parallel in 8 files via MPI-IO.

There is several considerations:

Does writing the output scale? When I wrote the routines (a few years ago), I tested writing into a single output file from 1000+ cores and got excellent performance (10s of GB/s). I don't see a problem with this.

Is this sensible for visualization? If you do visualization in serial, it doesn't matter. If you visualize in parallel, it might be helpful to have more than one file. With 8, you can at least run on 8 nodes. Maybe it makes sense to increase this number slightly? Not that it matters that much...

What if we used max(8,ceil(n_procs/128))? I mean, right now we write one file per processor; maybe we can find a compromise that at least makes sure we don't make this the one function that blocks everything if someone tried to run the program on 10k processors...

tjhei · 2019-11-02T21:55:55Z

I'm using the .visit file. I think it is true that Visit can now read .pvd files, right?

Yes, I verified that this works. Did you read my comment above? #8963 (comment)

bangerth · 2019-11-03T23:36:34Z

@tjhei: Thanks, I had missed that. Then let's not worry about .visit files any more from now on.

peterrum · 2019-11-05T22:39:59Z

@tjhei @bangerth I have implemented the suggestions. However, I have decided against max(8,ceil(n_procs/128)). Personally, I think everything above 1 is not really useful. This should be handled by a suitable configuration of MPI I/O according to the GPFS at hand. We show here an example, where we set the value to an arbitrary value (8), but actually it should be by default 1.

tjhei · 2019-11-05T22:47:24Z

Personally, I think everything above 1 is not really useful.

It should be helpful if you use parallel visualization (paraview server for example).

but actually it should be by default 1.

from the write performance point of view? Likely, yes.

@bangerth: maybe we can find a compromise that at least makes sure we don't make this the one function that blocks everything if someone tried to run the program on 10k processors...

I have to admit that I haven't made a performance comparison with the number of files and maybe we should test before making any more guesses (especially complicated ones)? I am fairly certain that 1 file is way more performant than anything else. I don't see why 8 files would "block everything" in a large run.

To summarize: IO is hard for large runs and us guessing is probably not a good strategy here. What do we want to achieve here: the best performance, something instructional, or something safe?

peterrum · 2019-11-05T22:51:56Z

It should be helpful if you use parallel visualization (paraview server for example).

I see! Thanks for the info!

peterrum · 2019-11-17T08:17:07Z

Anything missing?

masterleinad · 2019-11-30T16:33:32Z

/rebuild

masterleinad

Let's move forward here.

peterrum commented Oct 26, 2019

View reviewed changes

peterrum force-pushed the gridout_vtu_pvtu branch from 8907bbd to c30f23d Compare October 26, 2019 11:19

peterrum changed the title ~~Update tutorials to use write_vtu_with_pvtu_record [WIP]~~ Update tutorials to use write_vtu_with_pvtu_record Oct 26, 2019

tjhei reviewed Oct 26, 2019

View reviewed changes

peterrum force-pushed the gridout_vtu_pvtu branch from c30f23d to 45d2003 Compare October 26, 2019 14:18

tjhei reviewed Oct 26, 2019

View reviewed changes

examples/step-18/step-18.cc Outdated Show resolved Hide resolved

peterrum force-pushed the gridout_vtu_pvtu branch from 45d2003 to 18fefff Compare October 26, 2019 14:31

peterrum force-pushed the gridout_vtu_pvtu branch from 18fefff to 1f0acb4 Compare October 26, 2019 16:40

peterrum commented Oct 26, 2019

View reviewed changes

Update tutorials to use write_vtu_with_pvtu_record

11e19a1

peterrum force-pushed the gridout_vtu_pvtu branch from 1f0acb4 to 11e19a1 Compare October 28, 2019 21:14

tjhei added 3 commits October 29, 2019 07:51

remove dash

380f180

trailing dashes look ugly as the filename is "solution-_0001" etc.

pass cycle not name to function

29357ea

add a .visit master file with times

f491a28

bangerth reviewed Nov 2, 2019

View reviewed changes

Remove visit add comments

7095c31

masterleinad added ready to test Reviewed and ready to merge labels Nov 30, 2019

masterleinad approved these changes Nov 30, 2019

View reviewed changes

masterleinad merged commit c5626c5 into dealii:master Dec 1, 2019

-    // Let us open a file and write the data we have generated into it:
+    // Let us call a function that opens the necessary output files and writes the
+    // data we have generated into them. The function automatically constructs
+    // the file names from the given directory name (the first argument) and file
+    // name base (second argument). It augments the resulting string by pieces
+    // that result from the time step number and a "piece number" that corresponds
+    // to a part of the overall domain that can consist of one or more subdomains.
+    //
+    // The function also writes a record files (with suffix `.pvd`) for Paraview
+    // that describes how all of these output files combine into the data for
+    // this single time step:

	const auto pvtu_master_filename = data_out.write_vtu_with_pvtu_record(
	const std::string pvtu_master_filename = data_out.write_vtu_with_pvtu_record(

Update tutorials to use write_vtu_with_pvtu_record #8963

Update tutorials to use write_vtu_with_pvtu_record #8963

Conversation

peterrum commented Oct 25, 2019 • edited

Choose a reason for hiding this comment

tjhei commented Oct 26, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peterrum commented Oct 26, 2019

tjhei commented Oct 26, 2019

bangerth commented Oct 26, 2019 via email

peterrum commented Oct 26, 2019

tjhei commented Oct 26, 2019 • edited

peterrum Oct 26, 2019 • edited

Choose a reason for hiding this comment

peterrum commented Oct 28, 2019

tjhei commented Oct 29, 2019 • edited

peterrum commented Oct 29, 2019

peterrum commented Oct 31, 2019

tjhei commented Nov 1, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tjhei commented Nov 2, 2019

bangerth commented Nov 3, 2019

peterrum commented Nov 5, 2019

tjhei commented Nov 5, 2019 • edited

peterrum commented Nov 5, 2019

peterrum commented Nov 17, 2019

masterleinad commented Nov 30, 2019

masterleinad left a comment

Choose a reason for hiding this comment

peterrum commented Oct 25, 2019 •

edited

tjhei commented Oct 26, 2019 •

edited

peterrum Oct 26, 2019 •

edited

tjhei commented Oct 29, 2019 •

edited

tjhei commented Nov 5, 2019 •

edited