Add the final nonlinear residual to statistics file

[jdannberg]

We should add an option that writes the final nonlinear residual of each time step as output into the statistics file. This is useful for checking if there were any time steps where the nonlinear solver did not converge, and allows for easy plotting, e.g. in gnuplot.
It was brought up in #1983.

[bangerth]

Why not do it unconditionally? Who cares whether the stat file gains another column? :-)

[jdannberg]

I do, because it means I would have to update all of the scripts for plotting/tutorial slides, unless we always make it the last column.

[bangerth]

Hm, but that would mean that we can never add any new columns ever again :-( What do you use to create these plots?

[bangerth]

(I'm asking because we may be able to come up with a way to parse which column you need based on the header of the file...)

[jdannberg]

gnuplot, usually.
And I am not saying we can not add new columns, I just thought that we don't have to if we could also make them optional (after all, I don't know how many users would want to use that output, and the statistics file is already quite long and difficult to read).

[bangerth]

I'd just like to avoid that we have a parameter for each column in the stat file. Why should there be a parameter for the nonlinear residual but not for the number of inner M solver iterations? That seems silly.

[jdannberg]

Okay, but then I would suggest adding parameters for the things that are not needed most of the time instead of unconditionally adding even more things. After all, many of the columns are just added by postprocessors, so it's designed in a way that they are optional.

I find it difficult already to find a given column in that file (if I don't want to plot it but rather check if a certain value fits what I expect, such as for the test outputs), because there are more columns in one row than what fits on the screen.

[bangerth]

Would the following gnuplot command trickery help you?

gnuplot> TimeColumn = `grep 'Time (seconds)' statistics | perl -pi -e 's/\# *(\d+):.*/\1/g;'`
gnuplot> print TimeColumn
2
gnuplot> MinTempColumn = `grep 'Minimal temperature (K)' statistics | perl -pi -e 's/\# *(\d+):.*/\1/g;'`
gnuplot> print MinTempColumn
11
gnuplot> plot "statistics" using TimeColumn:MinTempColumn

The expressions are a bit complicated (and maybe I can make them easier to read), but at least the plot command becomes simpler to grasp.

If you think something like this would help with writing the plotting scripts in a more portable way, then I'd be happy to put an explanation of this sort of thing into the manual.

(Bonus points to anyone would has a simpler command than | perl -pi -e 's/\# *(\d+):.*/\1/g; to extract the column number from the beginning of the line...)

[tjhei]

(Bonus points to anyone would has a simpler command than | perl -pi -e 's/# (\d+):./\1/g; to extract the column number from the beginning of the line...)

grep -n "Minimal temperature" statistics | cut -f1 -d:

?

[bangerth]

Ha, I don't think I've ever heard of cut!

I think using grep -n works, but it's fragile because it assumed that column n is described in line n. Maybe this is better:

grep "Iterations for temperature solver" statistics | cut -d: -f1 | cut -c 3-

So, would something like this be useful in the manual?

[tjhei]

So, would something like this be useful in the manual?

Yes, I think so. Ideally we would use it in the various scripts.

We already use a different method in https://github.com/geodynamics/aspect/blob/master/benchmarks/blankenbach/get_statistics using gawk and sed.