DVGeometryVSP parallel finite difference has stochastic error on Ubuntu builds

[bbrelje]

Description

DVGeometryVSP parallel finite diff derivatives should have zero error compared to doing the same finite differences in serial. However, due to a bug in VSP, some processor's VSP models randomly have slightly different underlying parametric representations of each surface. The result is that doing parallel finite differences will sometimes produce a different Jacobian than doing the same in serial on the root processor.

DVGeoVSP test 2 documents the issue. The tolerance on the test, 5e-5, is much higher than we would like. We sometimes see errors in the 1e-5 or higher range. However, for unknown reasons, this only happens on Ubuntu-based containers. The CentOS images have zero error to machine precision.

Current behavior

Parallel finite differences using OpenVSP on Ubuntu creates random errors that are larger than we'd hope

Expected behavior

Parallel FD has zero error (to machine precision) compared to serial FD

Once this bug (on the OpenVSP side) is fixed the tolerance in test_2 should be tightened to 1e-16

[ramcdona]

Like my other comment -- if you can provide a MWE that produces a bug, I can spend some time trying to chase it down.

Since this appears to only occur on Ubuntu, I suspect a compiler related difference -- which probably means something with pointers, or un-initialized variables. We've made progress recently getting to the point of being AddressSanitizer and Valgrind clean, I am optimistic that this may already be better.