Boundary bug snippet

[MTCam]

Here's a "snippet" that reproduces our boundary issue (#280) hitting us in production runs.

The issue is reproduced when some partition(s) own 0 points on any of the boundaries. Oddly, I needed to run 4 ranks to reproduce the issue with this snippet. (I am unsure how both of 2 partitions end up owning points on all boundaries).

To reproduce the issue:
mpiexec -n 4 python -m mpi4py boundary-bug.py

I think the gist is summarized by this:

bnd_discr = discr.discr_from_dd(btag)
bnd_nodes = thaw(actx, bnd_discr.nodes())
bnd_normal = thaw(actx, discr.normal(btag))
result = bnd_nodes @ bnd_normal

... which fails when the partition owns 0 points on the btag boundary, apparently due to this difference in these data structures:

bnd_nodes=array([DOFArray((cl.Array([], shape=(0, 3), dtype=float64),)),
       DOFArray((cl.Array([], shape=(0, 3), dtype=float64),)),
       DOFArray((cl.Array([], shape=(0, 3), dtype=float64),))],
      dtype=object)
bnd_normal=array([DOFArray(()), DOFArray(()), DOFArray(())], dtype=object)

cc: @inducer , @anderson2981 , @majosm