Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a manifold to generate better meshes for the shell. #242

Merged
merged 3 commits into from
Jan 18, 2015

Conversation

tjhei
Copy link
Member

@tjhei tjhei commented Jan 12, 2015

This is a rework from #229, rebased to master so I can fix some problems.

@tjhei
Copy link
Member Author

tjhei commented Jan 12, 2015

(not ready to merge of course)

for (std::set<types::boundary_id>::iterator it = ids.begin();
it!=ids.end(); ++it)
if (*it > 1)
coarse_grid.set_boundary (*it, straight_border);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the place where I think a comment on why you're doing that (and which parts of the boundary it affects) is in place. I believe the word "border" only applies to the edge of a 2d object. Maybe call it straight_boundary?

@bangerth
Copy link
Contributor

Looks good with the one added piece of documentation mentioned in my last comment.

@tjhei
Copy link
Member Author

tjhei commented Jan 12, 2015

The problem was that your code crashed if you were running without a full shell:

An error occurred in line <8797> of file </root/deal.II/dealii/source/grid/tria.cc> in function
    const dealii::Boundary<dim, spacedim>& dealii::Triangulation<dim, spacedim>::get_boundary(dealii::types::manifold_id) const [with int dim = 2; int spacedim = 2; dealii::types::manifold_id = unsigned int]
The violated condition was: 
    man != NULL
The name and call sequence of the exception was:
    ExcMessage("You tried to get a Boundary, but I only have a Manifold.")
Additional Information: 
You tried to get a Boundary, but I only have a Manifold.

Stacktrace:
-----------
#0  /root/deal.II/installed/lib/libdeal_II.g.so.8.2.0: dealii::Triangulation<2, 2>::get_boundary(unsigned int) const
#1  /root/deal.II/installed/lib/libdeal_II.g.so.8.2.0: dealii::TriaAccessor<1, 2, 2>::get_boundary() const
#2  /root/deal.II/installed/lib/libdeal_II.g.so.8.2.0: void dealii::VectorTools::compute_nonzero_normal_flux_constraints<2, dealii::DoFHandler, 2>(dealii::DoFHandler<2, 2> const&, unsigned int, std::set<unsigned char, std::less<unsigned char>, std::allocator<unsigned char> > const&, dealii::FunctionMap<2, double>::type&, dealii::ConstraintMatrix&, dealii::Mapping<2, 2> const&)
#3  /root/deal.II/installed/lib/libdeal_II.g.so.8.2.0: void dealii::VectorTools::compute_no_normal_flux_constraints<2, dealii::DoFHandler, 2>(dealii::DoFHandler<2, 2> const&, unsigned int, std::set<unsigned char, std::less<unsigned char>, std::allocator<unsigned char> > const&, dealii::ConstraintMatrix&, dealii::Mapping<2, 2> const&)
#4  /aspect-tester/build-gcc/aspect: aspect::Simulator<2>::setup_dofs()
#5  /aspect-tester/build-gcc/aspect: aspect::Simulator<2>::run()
#6  /aspect-tester/build-gcc/aspect: main
--------------------------------------------------------

@tjhei
Copy link
Member Author

tjhei commented Jan 12, 2015

I also had to change the manifold id from 2 to something different, because this conflicted with the straight boundaries. Not sure if this is by design.

@tjhei
Copy link
Member Author

tjhei commented Jan 12, 2015

@tjhei
Copy link
Member Author

tjhei commented Jan 12, 2015

Do you have any suggestions?

@tjhei tjhei force-pushed the pull_229 branch 2 times, most recently from 3b43213 to 7df516f Compare January 15, 2015 01:47
@tjhei
Copy link
Member Author

tjhei commented Jan 15, 2015

it turns out that the two tests don't crash if we use MappingQ also in the interior!

@bangerth
Copy link
Contributor

Wow, who knew? How many iterations did you need for the first time step?

@tjhei
Copy link
Member Author

tjhei commented Jan 15, 2015

Wow, who knew? How many iterations did you need for the first time step?

a few more than with the old mesh:
tjhei@e96045e#diff-75

@tjhei
Copy link
Member Author

tjhei commented Jan 15, 2015

Sadly depth_average_01 (and 02 which is the same computation) are crashing now:
http://cdash.kyomu.43-1.org/testDetails.php?test=4717078&build=2556

What is weird about this is that the mesh is a unit cube refined to be 4x4 square cells so we should have a FlatManifold and it shouldn't make a difference. Switching back to the plain mapping in the interior or reducing the mapping from Q4 to Q1 fixes the problem.

Do you have any suggestion, @bangerth ? It might be a bug in deal.II, so maybe I need to sit down and construct a testcase...

@tjhei
Copy link
Member Author

tjhei commented Jan 15, 2015

I created a deal.II test and I can not find a difference between true and false for the interior mapping:
tjhei/dealii@c9fe51d

mapping quadrature points, shape_value, shape_gradient of FE_Q, and JxW are identical.

Any other ideas?

@bangerth
Copy link
Contributor

That's too crazy. Looking into it.

@bangerth
Copy link
Contributor

I'm seeing changes on the order of roundoff in the matrix. Need to look further, I assume...

@tjhei
Copy link
Member Author

tjhei commented Jan 15, 2015

So I now tried to circumvent the problem by using MappingQ(1,false) if we are using the BoxGeometry. This fixed the depth_average tests, but breaks
http://cdash.kyomu.43-1.org/testDetails.php?test=4826757&build=2613

ARGH!

@bangerth
Copy link
Contributor

So where do we stand now? I see differences on the order of roundoff in the matrix between using the higher order mapping everywhere or not. I think that's fair.

I did figure out which solver actually fails: it's the mass matrix solver (bottom right block) for which we use Trilinos's implementation of CG. I fail to see why this can happen -- the viscosity for this model is constant one.

Can you summarize what works and what doesn't? If I understand correctly, then this is the case:

  • use manifold stuff for spherical geometries works in principle but not for a testcase with very large viscosity variation (steinberger-viscosity.prm)
  • use manifold stuff plus always use higher order mappings makes steinberger-viscosity work but breaks depth-average.prm for non-obvious reasons.

@tjhei
Copy link
Member Author

tjhei commented Jan 16, 2015

So where do we stand now? I see differences on the order of roundoff in the matrix between using the higher order mapping everywhere or not. I think that's fair.

I did figure out which solver actually fails: it's the mass matrix solver (bottom right block) for which we use Trilinos's implementation of CG. I fail to see why this can happen -- the viscosity for this model is constant one.

Maybe the matrix is not exactly SPD?

Can you summarize what works and what doesn't?

A) use MappingQ(4,false): steinberger-viscosity and steinberger-viscosity-adiabatic crash (1)
B) use MappingQ(4,true): depth-average_01 and 02 crash (2)
C) use MappingQ(4,true) for shells and MappingQ(1,false) for box: periodic-box crashes (3)

@tjhei
Copy link
Member Author

tjhei commented Jan 16, 2015

I think we should keep MappingQ(4,true) for shells.

We could try our CG, which is probably more robust, or test with GMRES to see if that is the problem.

@tjhei
Copy link
Member Author

tjhei commented Jan 16, 2015

Even GMRES fails for the velocity block in periodic-box.prm. The example is periodic in x and has tangential conditions at the top/bottom so this is not too surprising. That the order of the mapping matters is just weird, though.

@bangerth
Copy link
Contributor

Yes, I forgot to mention this as well that using GMRES doesn't make a difference. I output the matrix before I left but didn't get to look at it. What could make it singular? It's a mass matrix on the pressure, after all, not a Laplace matrix on the velocities. I would like to understand what exactly is the problem here -- if it worked before, then it seems like that was only by accident.

I agree that we should keep Mapping(4,true). We could make it Mapping(4,false) if the domain indicates that it doesn't have curved cells. This would require some reshuffling of code, though, since we don't yet have the geometry by the time we create the mapping.

@bangerth
Copy link
Contributor

By the way, I use this patch to get the matrix in the step that fails:

diff --git a/source/simulator/solver.cc b/source/simulator/solver.cc
index 5d66526..37b4722 100644
--- a/source/simulator/solver.cc
+++ b/source/simulator/solver.cc
@@ -292,8 +292,11 @@ namespace aspect
             // if the solver fails, report the error from processor 0 with some additional
             // information about its location, and throw a quiet exception on all other
             // processors
-            catch (const SolverControl::NoConvergence &exc)
+            catch (const std::exception &exc)
               {
+                std::cout.precision(16);
+                stokes_preconditioner_matrix.block(1,1).print(std::cout, true);
+
                 if (Utilities::MPI::this_mpi_process(src.block(0).get_mpi_communicator()) == 0)
                   AssertThrow (false,
                                ExcMessage (std::string("The iterative (bottom right) solver in BlockSchurPreconditioner::vmult "

The matrix is only 25x25, so I plan on just putting it into maple or matlab or something and compute the spectrum. For reference, this is the matrix that fails:

Epetra::CrsMatrix
Number of Global Rows        = 25
Number of Global Cols        = 25
Number of Global Diagonals   = 25
Number of Global Nonzeros    = 169
Global Maximum Num Entries   = 9





Number of My Rows        = 25
Number of My Cols        = 25
Number of My Diagonals   = 25
Number of My Nonzeros    = 169
My Maximum Num Entries   = 9

   Processor    Row Index    Col Index           Value     
       0             0             0       69.44444444444441    
       0             0             1       34.72222222222222    
       0             0             2       34.72222222222221    
       0             0             3       17.36111111111111    
       0             1             0       34.72222222222222    
       0             1             1       138.8888888888888    
       0             1             2       17.36111111111111    
       0             1             3        69.4444444444444    
       0             1             4       34.72222222222222    
       0             1             5       17.36111111111111    
       0             2             0       34.72222222222221    
       0             2             1       17.36111111111111    
       0             2             2       138.8888888888887    
       0             2             3        69.4444444444444    
       0             2             6       34.72222222222221    
       0             2             7       17.36111111111111    
       0             3             0       17.36111111111111    
       0             3             1        69.4444444444444    
       0             3             2        69.4444444444444    
       0             3             3       277.7777777777774    
       0             3             4       17.36111111111111    
       0             3             5        69.4444444444444    
       0             3             6       17.36111111111111    
       0             3             7        69.4444444444444    
       0             3             8       17.36111111111111    
       0             4             1       34.72222222222222    
       0             4             3       17.36111111111111    
       0             4             4       138.8888888888888    
       0             4             5        69.4444444444444    
       0             4             9       34.72222222222226    
       0             4            10       17.36111111111114    
       0             5             1       17.36111111111111    
       0             5             3        69.4444444444444    
       0             5             4        69.4444444444444    
       0             5             5       277.7777777777774    
       0             5             7       17.36111111111111    
       0             5             8       69.44444444444439    
       0             5             9       17.36111111111114    
       0             5            10       69.44444444444451    
       0             5            13       17.36111111111114    
       0             6             2       34.72222222222221    
       0             6             3       17.36111111111111    
       0             6             6       138.8888888888888    
       0             6             7       69.44444444444439    
       0             6            15       34.72222222222225    
       0             6            16       17.36111111111113    
       0             7             2       17.36111111111111    
       0             7             3        69.4444444444444    
       0             7             5       17.36111111111111    
       0             7             6       69.44444444444439    
       0             7             7       277.7777777777774    
       0             7             8        69.4444444444444    
       0             7            15       17.36111111111113    
       0             7            16       69.44444444444446    
       0             7            17       17.36111111111114    
       0             8             3       17.36111111111111    
       0             8             5       69.44444444444439    
       0             8             7        69.4444444444444    
       0             8             8       277.7777777777774    
       0             8            10       17.36111111111114    
       0             8            13       69.44444444444449    
       0             8            16       17.36111111111114    
       0             8            17       69.44444444444444    
       0             8            21       17.36111111111116    
       0             9             4       34.72222222222226    
       0             9             5       17.36111111111114    
       0             9             9       138.8888888888888    
       0             9            10       69.44444444444436    
       0             9            11       34.72222222222229    
       0             9            12       17.36111111111114    
       0            10             4       17.36111111111114    
       0            10             5       69.44444444444451    
       0            10             8       17.36111111111114    
       0            10             9       69.44444444444436    
       0            10            10       277.7777777777774    
       0            10            11       17.36111111111114    
       0            10            12       69.44444444444451    
       0            10            13       69.44444444444434    
       0            10            14       17.36111111111115    
       0            11             9       34.72222222222229    
       0            11            10       17.36111111111114    
       0            11            11       69.44444444444441    
       0            11            12       34.72222222222229    
       0            12             9       17.36111111111114    
       0            12            10       69.44444444444451    
       0            12            11       34.72222222222229    
       0            12            12       138.8888888888889    
       0            12            13       17.36111111111115    
       0            12            14       34.72222222222229    
       0            13             5       17.36111111111114    
       0            13             8       69.44444444444449    
       0            13            10       69.44444444444434    
       0            13            12       17.36111111111115    
       0            13            13       277.7777777777774    
       0            13            14       69.44444444444449    
       0            13            17       17.36111111111116    
       0            13            21       69.44444444444441    
       0            13            22       17.36111111111116    
       0            14            10       17.36111111111115    
       0            14            12       34.72222222222229    
       0            14            13       69.44444444444449    
       0            14            14       138.8888888888889    
       0            14            21       17.36111111111116    
       0            14            22       34.72222222222231    
       0            15             6       34.72222222222225    
       0            15             7       17.36111111111113    
       0            15            15       138.8888888888889    
       0            15            16       69.44444444444441    
       0            15            18       34.72222222222225    
       0            15            19       17.36111111111114    
       0            16             6       17.36111111111113    
       0            16             7       69.44444444444446    
       0            16             8       17.36111111111114    
       0            16            15       69.44444444444441    
       0            16            16       277.7777777777775    
       0            16            17        69.4444444444444    
       0            16            18       17.36111111111114    
       0            16            19       69.44444444444449    
       0            16            20       17.36111111111115    
       0            17             7       17.36111111111114    
       0            17             8       69.44444444444444    
       0            17            13       17.36111111111116    
       0            17            16        69.4444444444444    
       0            17            17       277.7777777777774    
       0            17            19       17.36111111111115    
       0            17            20       69.44444444444446    
       0            17            21       69.44444444444451    
       0            17            23       17.36111111111117    
       0            18            15       34.72222222222225    
       0            18            16       17.36111111111114    
       0            18            18       69.44444444444443    
       0            18            19       34.72222222222229    
       0            19            15       17.36111111111114    
       0            19            16       69.44444444444449    
       0            19            17       17.36111111111115    
       0            19            18       34.72222222222229    
       0            19            19       138.8888888888889    
       0            19            20       34.72222222222229    
       0            20            16       17.36111111111115    
       0            20            17       69.44444444444446    
       0            20            19       34.72222222222229    
       0            20            20       138.8888888888889    
       0            20            21       17.36111111111117    
       0            20            23       34.72222222222235    
       0            21             8       17.36111111111116    
       0            21            13       69.44444444444441    
       0            21            14       17.36111111111116    
       0            21            17       69.44444444444451    
       0            21            20       17.36111111111117    
       0            21            21       277.7777777777775    
       0            21            22       69.44444444444453    
       0            21            23       69.44444444444443    
       0            21            24       17.36111111111118    
       0            22            13       17.36111111111116    
       0            22            14       34.72222222222231    
       0            22            21       69.44444444444453    
       0            22            22       138.8888888888888    
       0            22            23       17.36111111111118    
       0            22            24       34.72222222222232    
       0            23            17       17.36111111111117    
       0            23            20       34.72222222222235    
       0            23            21       69.44444444444443    
       0            23            22       17.36111111111118    
       0            23            23       138.8888888888889    
       0            23            24       34.72222222222234    
       0            24            21       17.36111111111118    
       0            24            22       34.72222222222232    
       0            24            23       34.72222222222234    
       0            24            24       69.44444444444466

Anyone good with matlab and wants to compute whether the matrix is SPD?

@ian-r-rose
Copy link
Contributor

Seems to be SPD to me, unless I've made a mistake.

Eigenvalues of that matrix:
[ 536.99426545 338.28613497 426.21322663 426.21322663 289.664209
289.664209 35.92240122 43.65830948 229.90695637 229.90695637
39.60191043 156.25 39.60191043 74.91912433 74.91912433
82.59304363 82.59304363 121.52777778 110.23633824 110.23633824
121.52777778 153.11519137 153.11519137 138.88888889 138.88888889]

I don't have a good intuition for the matrix, is the degeneracy expected?

@bangerth
Copy link
Contributor

No, it's not supposed to be degenerate. In fact, the eigenvalues look just right to me. On the other hand, we work on a square which has four-fold symmetry, so I would expect that there are four-fold degenerate eigenvalues, but I don't see any.

Here's another observation: the Trilinos solver only fails in this time step if the dst vector is nonzero. That's probably an oversight to begin with, given that we likely don't have a good initial guess for the CG solver to begin with. If I set the dst vector to zero, the solver succeeds (though I have to add that it then fails at a later step). I have no idea what that's supposed to indicate: an SPD matrix has full image and range spaces, so it's not like the solver might fail for some initial guesses (not for some particular right hand sides, for that matter). I guess I have to construct a small example out of this that simply tries out AztecOO separately :-(

@tjhei
Copy link
Member Author

tjhei commented Jan 16, 2015

I agree that we should keep Mapping(4,true). We could make it Mapping(4,false) if the domain indicates that it doesn't have curved cells. This would require some reshuffling of code, though, since we don't yet have the geometry by the time we create the mapping.

I already did that yesterday: tjhei@b18093e

@bangerth
Copy link
Contributor

My last comment corresponds to the has_curved_cell patch.

@tjhei
Copy link
Member Author

tjhei commented Jan 16, 2015

okay, periodic_box.prm doesn't crash in the A solve when I change the coarse grid solver of AMG from KLU to "ML symmetric Gauss-Seidel". That is probably a bad idea for performance, but this shows us what the problem is:
I assume ML coarsens in a way that KLU fails immediately or fails to provide a SPD operator. @kronbichler is there a way to tune the coarse grid solver and apply something like diagonal strengthening or something?

@bangerth
Copy link
Contributor

Wait, what? I missed the periodic-box thing. What's going wrong there? Is the A matrix singular for this case? I think we really do want the direct KLU solver on the coarse mesh, though, no?

@tjhei
Copy link
Member Author

tjhei commented Jan 16, 2015

Wait, what? I missed the periodic-box thing. What's going wrong there? Is the A matrix singular for this case?
Please scroll up and read my comments. A is singular because of the periodicity, yes. It used to work in the past, though (out of luck?). Maybe we need to change this...

I think we really do want the direct KLU solver on the coarse mesh, though, no?

I think so, yes. I just did this to see what is happening in this test.

@tjhei tjhei mentioned this pull request Jan 16, 2015
@kronbichler
Copy link
Contributor

@tjhei Regarding the KLU: From the ML guide, I can't see a way to set coarse solver parameters with respect to that (other than defining one's own solver). On the other hand, I can't think of KLU to not do the 'right' thing when it gets a singular matrix and failing only in case the incoming vector is not in the range of the matrix.
But I don't really understand: The solver for the pressure mass matrix fails (which is weird given the above eigenvalues), but why is there a KLU involved? Are you using AMG on the pressure mass matrix?

@bangerth
Copy link
Contributor

@kronbichler : The case @tjhei asks about is for the Laplace matrix which is singular for this case. The mass matrix is a different testcase.

@tjhei
Copy link
Member Author

tjhei commented Jan 16, 2015

@kronbichler No, if I play around with the MappingQ a different test fails. The velocity-velocity block Trilinos::SolverCG with AMG fails with

    AztecOO::Iterate error code -2: " "numerical breakdown"

If I switch to a different coarse solver, it works.

@bangerth
Copy link
Contributor

I've isolated the depth_average failure into this deal.II testcase:
dealii/dealii#448
It definitely sounds like a Trilinos problem.

I think we should merge this PR (after some cleanup of the commit history) to mainline and deal with the remaining periodic-box at a later time.

@tjhei tjhei force-pushed the pull_229 branch 2 times, most recently from 81544cd to af3a54b Compare January 17, 2015 03:43
bangerth and others added 3 commits January 16, 2015 23:06
This only works with a sufficiently new deal.II, of course.
- set straight boundary for quarter/half shell walls
- only set boundary objects as needed for spherical shell
- remove deal.II version check
- use higher order mapping also in the interior
@tjhei
Copy link
Member Author

tjhei commented Jan 17, 2015

I cleaned up the commits. Are you okay with the code changes?

Before we merge I want to a) read through the test changes, and b) wait for the tester. Tomorrow...

@bangerth
Copy link
Contributor

Yea, I like it. Thanks for working it over -- merge whenever the tests are to your liking!

tjhei added a commit that referenced this pull request Jan 18, 2015
Use a manifold to generate better meshes for the shell.
@tjhei tjhei merged commit eef6299 into geodynamics:master Jan 18, 2015
@tjhei tjhei deleted the pull_229 branch January 18, 2015 22:37
@bangerth
Copy link
Contributor

What an ordeal :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants