AMR w/ TET14s

[nmnobre]

Hi Roy (@roystgnr),

I've been trying to put together support for TET14s in MOOSE, see https://github.com/farscape-project/moose/tree/tet14.
Now, as part of the tests, I basically tried to replicate what's already there for TET10s, including for AMR.

So, if I do:

./moose_test-opt -i tests/mesh/high_order_elems/high_order_elems.i Mesh/uniform_refine=1 Mesh/elem_type=TET14 Variables/u/order=SECOND Mesh/dim=3 Mesh/nx=1 Mesh/ny=1 Mesh/nz=1

I get a deficient 4th child for the first TET:

Elem Information
   id()=28, unique_id()=554, subdomain_id()=0, processor_id()=0
   type()=TET14
   dim()=3
   n_nodes()=14
   mapping=LAGRANGE_MAP
    0  Node id()=4, processor_id()=0, Point=(x,y,z)=(    0.25,     0.25,        0)
    DoFs=(0/0/1) 
    1  Node id()=8, processor_id()=0, Point=(x,y,z)=(     0.5,      0.5,     0.25)
    DoFs=(0/0/12) 
    2  Node id()=6, processor_id()=0, Point=(x,y,z)=(       0,      0.5,        0)
    DoFs=(0/0/2) 
    3  Node id()=7, processor_id()=0, Point=(x,y,z)=(    0.25,     0.25,     0.25)
    DoFs=(0/0/3) 
    4  Node id()=138, processor_id()=0, Point=(x,y,z)=(   0.375,    0.375,    0.125)
    DoFs=(0/0/16) 
    5  Node id()=138, processor_id()=0, Point=(x,y,z)=(   0.375,    0.375,    0.125)
    DoFs=(0/0/16) 
    6  Node id()=126, processor_id()=0, Point=(x,y,z)=(   0.125,    0.375,        0)
    DoFs=(0/0/5) 
    7  Node id()=129, processor_id()=0, Point=(x,y,z)=(    0.25,     0.25,    0.125)
    DoFs=(0/0/8) 
    8  Node id()=155, processor_id()=0, Point=(x,y,z)=(   0.375,    0.375,     0.25)
    DoFs=(0/0/28) 
    9  Node id()=130, processor_id()=0, Point=(x,y,z)=(   0.125,    0.375,    0.125)
    DoFs=(0/0/9) 
    10  Node id()=165, processor_id()=0, Point=(x,y,z)=(    0.25, 0.416667, 0.0833333)
    DoFs=
    11  Node id()=11, processor_id()=0, Point=(x,y,z)=(0.333333, 0.333333, 0.166667)
    DoFs=
    12  Node id()=166, processor_id()=0, Point=(x,y,z)=(    0.25, 0.416667, 0.166667)
    DoFs=
    13  Node id()=134, processor_id()=0, Point=(x,y,z)=(0.0833333,     0.25, 0.0833333)
    DoFs=

The 5th node is the same as the 4th node, so that's already alarming, but the 13th node is also wrong. So, then, after some digging into the code, I came across the embedding matrix for the 4th child of TET14s:

libmesh/src/geom/cell_tet14.C

Lines 581 to 598 in 52f92e5

	// embedding matrix for child 4
	{
	// 0 1 2 3 4 5 6 7 8 9 10 11 12 13
	{ 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.}, // 0
	{ 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.}, // 1
	{ 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.}, // 2
	{ 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.}, // 3
	{-.03125,.09375, 0.,-.03125,0.125, 0., 0.,-0.125, 0.125, 0., 0.,.84375, 0., 0.}, // 4
	{-.03125,.09375, 0.,-.03125,0.125, 0., 0.,-0.125, 0.125, 0., 0.,.84375, 0., 0.}, // 5
	{.09375,-.03125,-.03125, 0., 0.125,-0.125, 0.125, 0., 0., 0.,.84375, 0., 0., 0.}, // 6
	{.09375,-.03125, 0.,-.03125,0.125, 0., 0., 0.125,-0.125, 0., 0.,.84375, 0., 0.}, // 7
	{-.03125,-.03125, 0.,.09375,-0.125, 0., 0., 0.125, 0.125, 0., 0.,.84375, 0., 0.}, // 8
	{.09375, 0.,-.03125,-.03125, 0., 0., 0.125, 0.125, 0.,-0.125, 0., 0., 0.,.84375}, // 9
	{ 2/r72, 2/r72, 0., 0., 0.,-2/r18,-2/r18,-2/r18,-2/r18,-2/r18, 0.5, 0.5, 0.25, 0.25}, // 10
	{ 0., 0, 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.}, // 11
	{ 2/r72, 0., 0., 2/r72,-2/r18,-2/r18,-2/r18, 0.,-2/r18,-2/r18, 0.25, 0.5, 0.25, 0.5}, // 12
	{ 5/r18, 0.,-1/r18,-1/r18, 0., 0., 4/r18, 4/r18, 0.,-2/r18, 0., 0., 0., 0.5} // 13
	},

And, sure enough, the lines for nodes 4 and 5 are (erroneously) the same. So, I figured, let's fix that, see https://github.com/farscape-project/libmesh/tree/tet14_amr (I've also changed node 0, just to make sure I wasn't crazy and that I'm indeed using a new matrix), and........

Elem Information
   id()=28, unique_id()=555, subdomain_id()=0, processor_id()=0
   type()=TET14
   dim()=3
   n_nodes()=14
   mapping=LAGRANGE_MAP
    0  Node id()=165, processor_id()=0, Point=(x,y,z)=(   19.25,    19.25,        0)
    DoFs=(0/0/34) 
    1  Node id()=8, processor_id()=0, Point=(x,y,z)=(     0.5,      0.5,     0.25)
    DoFs=(0/0/12) 
    2  Node id()=6, processor_id()=0, Point=(x,y,z)=(       0,      0.5,        0)
    DoFs=(0/0/2) 
    3  Node id()=7, processor_id()=0, Point=(x,y,z)=(    0.25,     0.25,     0.25)
    DoFs=(0/0/3) 
    4  Node id()=138, processor_id()=0, Point=(x,y,z)=(   0.375,    0.375,    0.125)
    DoFs=(0/0/16) 
    5  Node id()=138, processor_id()=0, Point=(x,y,z)=(   0.375,    0.375,    0.125)
    DoFs=(0/0/16) 
    6  Node id()=126, processor_id()=0, Point=(x,y,z)=(   0.125,    0.375,        0)
    DoFs=(0/0/5) 
    7  Node id()=129, processor_id()=0, Point=(x,y,z)=(    0.25,     0.25,    0.125)
    DoFs=(0/0/8) 
    8  Node id()=155, processor_id()=0, Point=(x,y,z)=(   0.375,    0.375,     0.25)
    DoFs=(0/0/28) 
    9  Node id()=130, processor_id()=0, Point=(x,y,z)=(   0.125,    0.375,    0.125)
    DoFs=(0/0/9) 
    10  Node id()=166, processor_id()=0, Point=(x,y,z)=(    0.25, 0.416667, 0.0833333)
    DoFs=
    11  Node id()=11, processor_id()=0, Point=(x,y,z)=(0.333333, 0.333333, 0.166667)
    DoFs=
    12  Node id()=167, processor_id()=0, Point=(x,y,z)=(    0.25, 0.416667, 0.166667)
    DoFs=
    13  Node id()=134, processor_id()=0, Point=(x,y,z)=(0.0833333,     0.25, 0.0833333)
    DoFs=

The coordinates for nodes 5 and 13 didn't change at all????
I know you do some caching, but I don't really understand how it works...

I'm just very confused...
Cheers,
Nuno (cc @karthichockalingam)

[nmnobre]

The bracketing pairs are also repeated for nodes 4 and 5... Right... that might be it?

[roystgnr]

That's got to be it. TET14 is one of those elements complicated enough that I gave up on trying to generate the embedding matrices by hand, and wrote a little code to get them from the bracketing nodes. So if I screwed up a bracketing node pair it'd reflect in a screwed up embedding matrix.

It's a disaster that this isn't showing up in any unit tests, though. I guess the fe_test.h code never uses refinement, and the elem_test.C test_refinement() never instantiates an FE, but I'd have hoped even the topology-only tests in test_refinement() might have caught a duplicated node location. I guess not...